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Foreword 


Evaluation systems face many challenges. In addition to the quality, rele- 
vance and timeliness of the evaluation itself, a major challenge lies in conveying 
the evaluation results to multiple audiences both inside and outside develop- 
ment agencies. Thus feedback and communication of evaluation results are inte- 
gral parts of the evaluation cycle. Effective feedback contributes to improving 
development policies, programmes and practices by providing policymakers with 
the relevant evaluation information for making informed decisions. 

The origin of the Working Party on Aid Evaluation's (WP-EV) attention to feed- 
back issues stems from the 1998 "Review of the DAC Principles for Evaluation of 
Development Assistance". This report highlighted a concern widespread in DAC 
Members' central evaluation offices about the adequacy of current practices for 
disseminating lessons from evaluations. In a subsequent document, "Complement- 
ing and Reinforcing the DAC Principles for Aid Evaluation" (19981, it was concluded 
that feedback and communication practises need to be improved, and that evalu- 
ation results should be used more actively in order to enhance current and future 
aid policies and programmes. 

To take this work forward the Japanese Government hosted a DAC WP-EV 
workshop in Tokyo on 26-28 September 2000 entitled "Evaluation Feedback for 
Effective Learning and Accountability". The purpose of this conference was to 
share experiences among WP Members and other participants and to draw lessons 
from current practices in the area of feedback mechanisms. 

This publication is composed of two parts: The Workshop Report, based on the 
fore-mentioned meeting, highlights the various issues raised, topics of discussion 
and different feedback systems, and outlines the areas identified by participants 
as most relevant for improving evaluation feedback. The Background Synthesis 
Report, intended as a starting point for discussion at the workshop, outlines the 
main concerns and challenges facing evaluation feedback and the means to 
address these. The report is based on an analysis of questionnaire results, and a 
review of previous initiatives in this area. 

I would like to thank the Japanese Government, and in particular the evaluation 
departments of the Ministry of Foreign Affairs, the Japan International co-operation 
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Agency (IICA) and )apan Bank for International co-operation (JBIC) for hosting the 
workshop, and for their financial support for the conference and this publication. I 
am also grateful to members of the Steering Group who helped prepare the event, 
and to workshop participants for contributing quality presentations and papers. 
The Workshop Report draws heavily on participants' individual contributions, 
including power point presentations, papers and speeches. The Institute of 
Development Studies (IDS), and in particular Geoff Barnard (Head of Information, 
Project Manager) and Catherine Cameron (Senior Consultant), was commissioned 
by Japan to prepare the reports and I would like to express my appreciation to 
both authors for their contributions. 


Niels Dabelstein 

Chair of the Working Party on Aid Evaluation 


The Development Assistance Committee (DAC) Working Party on Aid Evaluation 
is an international forum where bilateral and multilateral development evaluation 
experts meet periodically to share experience to improve evaluation practice and 
strengthen its use as an instrument for development co-operation policy. 

It operates under the aegis of the DAC and presently consists of 30 represen- 
tatives from OECD Member countries and multilateral development agencies 
(Australia, Austria, Belgium, Canada, Denmark, European Commission, Finland, 
France, Greece, Ireland, Italy, Germany, )apan, Luxembourg, the Netherlands, 
New Zealand, Norway, Portugal, Spain, Sweden, Switzerland, United Kingdom, 
United States, World Bank, Asian Development Bank, African Development Bank, 
Inter-American Development Bank, European Bank for Reconstruction and 
Development, UN Development Programme, International Monetary Fund, plus 
two non-DAC Observers, Mexico and Korea). 

Further information may be obtained from Hans Lundgren, Advisor on Aid 
Effectiveness, OECD, Development Cooperation Directorate, 2, rue Andre-Pascal, 
75775 Paris Cedex 16, France. Website: www.oecd.org/dac/evalmtion. 
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Part I 

WORKSHOP REPORT 



Evaluation Feedback for Effective Learning 
and Accountability 

Tokyo, 26-28 September 2000 


Executive summary 

1 . There was a keen recognition among all the agencies present at the Workshop 
of the need to improve their evaluation feedback practices. However the dif- 
ferences between agencies in their background, structure and priorities 
means that this is not an area where a blueprint approach is appropriate. 

2. Several important challenges were laid down at the Workshop, in particular 
the need for much fuller involvement of developing country partners in the 
evaluation and learning process, and the importance of identifying target 
audiences more carefully and becoming more demand led, rather than sup- 
ply driven. 

3. There were differences of opinion on the relationship between learning and 
accountability, and the pros and cons of evaluation units guarding their inde- 
pendent role. Some agencies argued that the accountability that comes from 
full independence is an essential foundation for learning. Others were of the 
view that too much emphasis on independence can be counterproductive if it 
gets in the way of the partnerships needed to facilitate learning. 

4. There was agreement, however, that this is not an either/or situation. Both 
accountability and learning are important goals for evaluation feedback. Of 
the two, learning is the area where agencies recognise the greatest chal- 
lenges. 

5. There are many obstacles to learning relating to organisational culture, lack of 
incentives to learn, unequal aid relationships and other factors. These need 
to be identified and specifically targeted if progress is to be made. 

6. Discussion of different stakeholder groups emphasised the need to tailor 
feedback approaches to suit different target audiences. This is a clear case 
where one size does not fit all. 

7. Despite the risks involved, the case for full disclosure of evaluation reports 
was widely accepted. Relations with parliaments and the media, for example, 
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hinge on trust. This requires an atmosphere of openness and a willingness 
to discuss problems as well as success stories. 

8. Getting evaluation lessons across to a range of diverse audiences requires a 
sophisticated and concerted approach, particularly given the information 
overload affecting many. The importance of relevance, timeliness, brevity and 
quality were stressed. Moving from project evaluations to a higher level coun- 
try or thematic focus was seen as an important way of increasing impact, par- 
ticularly when evaluations can be timed to coincide with major policy reviews. 

9. Internet-based communication tools are seen as an important addition to the 
evaluation toolbox, though they need to be used intelligently and are not a 
replacement for conventional approaches. While acknowledging the potential 
offered by new communication approaches, the value of "learning by doing" 
and of face-to-face dialogue is still widely recognised. 

10. Involving developing country partners in evaluation and lesson learning is 
seen as a major priority by many agencies, as most recognise that current 
practices fall far short of what is required for meaningful partnerships. Partici- 
pation and dialogue needs to start right at the beginning of the project cycle 
rather than getting tacked on at the end, or squeezed out because of time con- 
straints. Language issues and capacity building also need to be addressed much 
more seriously than in the past. 

1 1. With new trends in aid delivery, the need for effective donor coordination is 
more urgent than ever. This points to the need for more joint evaluations and 
for much greater attention being given to lesson sharing between agencies. 

12. The status of evaluation units is a big determinant of their impact within an 
organisation. For evaluation to become a key focus for learning and account- 
ability, evaluation units need to have clear backing from senior management. 

13. Evaluation units will need an injection of new skills, particularly in communi- 
cation work, if they are to improve their feedback practices. Lack of resources, 
however, was not seen to be a major constraint since the case for better feed- 
back is strong. 

1 4. Although it was accepted that a standardised approach to improving evaluation 
feedback would not be appropriate, a number of areas for action were identi- 
fied at various levels (see Box 2). 

15. The overall consensus was that in relation to evaluation feedback the challenge 
for development agencies is to do more and do it better. 



1 . Introduction 


The need for improved evaluation feedback is widely recognised by develop- 
ment agencies, both on grounds of accountability and learning. Greater account- 
ability is seen as a prerequisite for continued support for development assistance 
within donor countries, while also being a vital component in creating robust and 
meaningful partnerships with countries and organisations that are recipients of 
aid. Better learning is crucial for improving the effectiveness of aid and ensuring 
that the hard won lessons from experience - both positive and negative - are 
heeded. Evaluation feedback is an umbrella term describing the various channels and 
mechanisms by which these crucial messages get through - and make a difference. 

The Tokyo Workshop "Evaluation Feedback for Effective Learning and 
Accountability" (EFELA), held in September 2000, was convened by the DAC 
Working Party on Aid Evaluation in order to focus in on this important topic. The 
Workshop brought together representatives from 22 donor countries and multilat- 
eral development agencies, and participants from 6 partner countries. Lasting two 
and a half days it consisted of a mix of plenary sessions where agencies gave short 
presentations on their experience, and a series of smaller working groups looking 
at particular themes in more detail. The highlights of the discussion at the work- 
shop are summarised and included in the Press Release by Prof. Ryokichi Hirono, 
Chairman of the DAC Tokyo Workshop and Mr. Niels Dabelstein, Chairman of the 
DAC Working Party on Aid Evaluation. A full programme, list of participants and 
the press release is provided in Annexes 1 , 2 and 3. 

In preparation for the Workshop, the Institute of Development Studies (IDS), 
at the University of Sussex, was commissioned by Japan to prepare a Synthesis 
Report. This provided information on agencies’ current feedback practices and an 
insight into the issues that Working Party Members see as being most crucial in 
improving evaluation feedback. 1 

This Workshop Report builds on this earlier document and should be read in 
conjunction with it. This Report seeks to highlight the main points raised during 


1. The Synthesis report was based on responses to a background questionnaire sent to all 
Working Party Members, and follow-up interviews with a cross-section of agencies. 


© OECD 2001 



Evaluation Feedback for Effective Learning and Accountability 


the Workshop and draw out the areas of consensus that emerged, as well as the 
differences in approach and outlook between agencies. 

These differences were one of the underlying features of the Workshop. As 
was pointed out in the earlier Synthesis Report, agencies are not all starting from 
the same place in terms of their approach to evaluation feedback, their audiences, 
their relative priorities, or the institutional context in which they work. So while 
everyone at the Workshop agreed on the urgent need to improve evaluation feedback 
it was clear that no one approach, or set of approaches, would suit all agencies. As one 
participant from China put it, this is a case where " one medicine cannot cure all patients." 
This point was reinforced by a colleague from the Philippines who commented that 
“we shouldn't be looking for prescriptions ... it’s outcomes that count". 

Discussions at the Workshop ranged over a wide area and tackled the subject 
of evaluation feedback from a number of different angles. This Report is grouped 
under a series of broad headings that correspond to the principle areas of debate. 



2. The Changing Aid Landscape 


The Workshop took place against the background of a rapidly evolving aid 
landscape. Change is nothing new to the development assistance business. But at 
present there is a sense that the rate and extent of change is greater than it has 
been for many years. Some of the aspects that were highlighted include: 

• New agendas - as Norway observed, development agencies face chal- 
lenges caused by the crowded policy agenda. There are many elements 
to this including the need to address issues such as conflict reduction, 
anti-corruption and governance. Many of these suffer from definitional 
problems and unclear time horizons, which makes them intrinsically diffi- 
cult to evaluate. 

• New mechanisms - there are also a whole range of new aid mechanisms 
being tried. These include Sector Wide Approaches, the Comprehensive 
Development Framework, Poverty Reduction Strategy Papers, Poverty 
Reduction and Growth Frameworks, Medium Term Expenditure Frame- 
works, and the Sustainable Livelihoods Approach. Some of these imply 
fundamental changes in the way donors work, requiring significant shifts 
in how evaluation is approached. 

• New partners - development agencies are finding themselves working 
with new partners and in different partnership configurations. Multi-donor 
consortia are becoming more common, and agencies are moving 
beyond government-to-government links to forge direct links with civil 
society. Some brand new partners are also emerging, such as the military, 
which is becoming a key partner for donors working in post-conflict or 
conflict resolution situations. This may mean working with UN-led forces, 
or developing working relationships with Ministries of Defence for inter- 
ventions in countries such as Bosnia, Sierra Leone, and East Timor. 1 

• New pressures - the growing influence of international pressure groups is 
adding a new dimension to aid debates, as demonstrated in the recent 


1. In the UK, the Ministry of Foreign Affairs, DFID and the Ministry of Defence have agreed 

to pool money into a joint conflict prevention budget to maximise effort. 15 | 
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Seattle and Prague protests. Often focusing on single issues, these groups 
have become highly skilled in gaining international media attention and 
using the Internet to organise and mobilise. These groups may not be 
elected, representative or accountable, in a conventional sense, but they 
have become increasingly powerful and certainly cannot be ignored. 

• New horizons - aid delivery has been shifting from project, to pro- 
gramme, to country level. But there is now an array of issues on the 
agenda which require a regional or global approach, for example, issues 
such as trade, migration, climate change, genetically modified organisms, 
and HIV/AIDS. 

• New Emphasis on Aid Effectiveness - The recent emphasis on aid effective- 
ness is not only due to increased demand for better development results 
and accountability, but is also partly a result of budgetary constraints. 

Agencies are having to adapt to these changes and this is creating many new 
challenges for evaluation units, requiring them to think and work in new ways. Discus- 
sion of evaluation feedback needs to be considered within this evolving context, 
learning from the past, but also looking to the future of where evaluation is heading. 



3. The Feedback Challenge 


3. 1 . Learning and accountability 

There was much discussion of the relationship between these dual goals of 
evaluation feedback, and their relative importance. While it was recognised that 
there is significant overlap between them, it was also seen that they are not iden- 
tical, involving different target audiences and requiring sometimes quite different 
approaches. A range of views were expressed on how the two functions relate: 

• Some agencies argued that accountability is still the core function of central 
evaluation units, and that the new emphasis on learning needs to build 
from this and not be seen as being in opposition to it. Separating them 
out, they felt, was creating a "false dichotomy". 

• Others see it differently and pointed out the tensions that can arise 
between the two; IFAD, for example, is now putting learning explicitly at 
the top of its agenda. 

• SIDA suggested that it was useful to decide on a case-by-case basis 
whether learning or accountability was the priority in a particular evaluation. 
This would then affect the design of the evaluation and the role of stake- 
holders. 

• The World Bank suggested that accountability should create the "incentive 
framework for learning’’. 

• From a partner country perspective, Lao PDR argued that learning and 
accountability are two sides of the same coin. Bangladesh commented 
that for them learning and accountability are a little different in practice, 
since learning is based in the Ministry of Planning implementation and 
monitoring unit, and accountability is more the preserve of the Auditor 
General. Therefore learning may not be owned across the system and 
may not translate into future activities. 

• )apan emphasised the importance of establishing an effective "Feedback 
Mechanism" to senior policy makers, senior evaluation department and 
operational department staffs. 
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These contrasting attitudes are partly a reflection of the differing backgrounds 
of agencies and the relative positioning of evaluation units within them. Evalua- 
tion units reporting directly to governing boards or to parliament, for example, are 
clearly in a different position compared to those more closely linked with policy or 
operational units. This has an important bearing on which audiences or stakeholders 
are de facto the highest priority, and what kinds of feedback tend to take precedence. 

While there are differences in approach, it was agreed that this is not an either/or 
situation - both learning and accountability are vital. Of the two, however, learning is 
the area in which most agencies see the greatest need for improvement. Accountabil- 
ity has long been a core concern for evaluation units; learning is now the "new frontier" 
where a whole range of new opportunities and challenges exist. 

3.2. The learning frontier 

Concepts of knowledge management and "learning organisations" borrowed from 
the corporate sector have made considerable inroads into development agency think- 
ing recently, although approaches and terminology vary. The Workshop considered 
how these concepts can be applied in an evaluation setting, recognising that evalua- 
tion units do not have a monopoly on knowledge. Compared to other departments, 
however, it was pointed out that they do have the advantage of being one of the most 
systematic in their approach, and having often the most comprehensive coverage. 
They therefore have an important role to play as central knowledge holders and a sig- 
nificant stake in how new thinking on knowledge and learning evolves. 

A number of agencies stressed the need to "unpack" the language on learning 
and to clarify what is meant by it. Learning at the individual level, for example, is 
quite different than learning at the organisational level. There are also different 
stages in the learning process - from finding new results, to validating and verify- 
ing them, transferring them to others, and codifying them into guidelines. The 
World Bank contrasted three types of learning - learning from "doing", from 
research, and from evaluation - and noted that mechanisms need to be in place to 
allow effective learning from sources outside the institution, as well as within it. France 
noted that evaluation reports are just one component in decision making, comment- 
ing that "if it were so simple, our work would have been over a long time ago". 

If evaluation units are to take on a proactive learning role they need to 
develop a more sophisticated view of how learning takes place within and 
between organisations, and where the bottlenecks arise. Several agencies are 
already addressing this challenge (see Box 1: Signs of improving conditions for 
evaluation-based learning in German aid agencies). AusAID is undertaking a study 
to understand how their staff and consultants learn, as part of a wider exercise 
revising their lessons learned database. The study will be looking at formal and 
I 18 informal systems for knowledge management, information sharing and lesson 
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Box 1 . Signs of improving conditions for evaluation-based learning 
in German aid agencies' 

• Germany’s diversified development co-operation structure is now gradu- 
ally moving towards greater concentration on particular issues, priority 
areas and countries. There is also a parallel trend towards greater decen- 
tralisation. 

• German official aid agencies see themselves more than ever as learning 
organisations , and are beginning to restructure their management systems 
accordingly. Evaluation systems are intended to play a key part in this, and are 
being given greater priority and greater institutional independence. 

• The quality of evaluation is improving. More sophisticated methods, more 
impact orientation and a greater number of broader-based evaluations I not 
confined to a single project} all offer the prospect that in future more of the 
knowledge will be generated that is needed for both quality improvement 
and conceptual advancement of development co-operation work, and for 
greater external accountability. 

• Aid agencies themselves believe it is important to increase the extent to 
which they systematise and institutionalise their feedback system for evalua- 
tion-based learning and accountability. 

• The organisations are making increasing use of modern, intranet-based infor- 
mation and documentation (I&D) systems . These reduce information gathering 
and search costs, and help to establish an institutional memory. However, the 
functionality of l&D systems leaves room for improvement. In particular, 
efforts are needed to manage the growing flood of information. Further 
problems, given the diversity of institutions, are how to network the l&D 
systems and how to cater for upward feedback in heavily decentralized 
organisations. 

• Aid agencies see a strong need to do more to promote the internalisation of 
evaluation lessons, taking a more systematic and innovative approach. Some 
are currently appraising the inclusion of this in an overall system of knowl- 
edge management. 

• Implementation monitoring (i.c., implementation of evaluation lessons| is 
essentially based on established M&E systems. 

• The foundation of the Deutsche Gesellschaft fur Evaluation (German Society 
for Evaluation) and the annual evaluation meetings now staged by the BMZ 
with the major German aid agencies have both given a substantial boost to 
horizontal learning among German aid agencies in recent years. 

• By putting its evaluation reports on public release, the BMZ has initiated a 
turn in the tide for German aid agencies’ public disclosure policy Numerous 
agencies have said they intend to follow, or at least look into the possibility 
of following the Ministry’s example. Others remain hesitant to make this 
move. 
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Box 1 . Signs of improving conditions for evaluation-based learning 
in German aid agencies (cont . I 

• Most German development aid organisations have in the past largely 
addressed their feedback to internal audiences. The media and general pub- 
lic have not been a high priority for them as target audiences. In-country part- 
ners and stakeholders still tend to be passed over too often in the feedback 
process. However, the need for greater participation is generally recognized. 

• Given that Germany's aid organisations usually have well-developed project 
controlling and M&E systems, feeding back evaluation findings into on-going 
projects is relatively straightforward. In contrast to this, utilising the lessons 
learned from broader-based evaluations is a considerably more challenging task. 


* Adapted from "Evaluation Based Learning - The German Case", German Federal Ministry 
for Economic Co-operation and Development, study submitted to the DAC Working Party 
on Aid Evaluation (WP-EV) Workshop, Tokyo 26-28 September 2000. 


learning across a range of functions, not just evaluation. Once they have a better 
understanding of this it will be possible to consider the format, design and modes 
of communication to be developed for evaluation feedback. CIDA has a Continuous 
Learning Division which works with the Performance Review Branch to develop 
best practice modules and ask the question "how are we learning?” It also has a 
Corporate Memory Unit, which together with a contractor is working to organise 
the lessons learned into a coherent, subject-based, easily retrievable database. 

3.3. Obstacles to learning 

There are many obstacles that can get in the way of learning. Those brought 
up at the Workshop include: 

• Organisational culture - some organisations have a culture where 
accountability tends to be associated with blame. This has the effect of 
discouraging openness and learning. In others, it is more acceptable to 
own up to mistakes and see these as opportunities for learning, recognis- 
ing that there is often as much to learn from poorly performing projects as 
there is from success stories. 

• Pressure to spend - learning takes time, and pressure to meet disburse- 
ment targets can lead to shortcuts being taken during project planning 
and approval stages, with lessons from previous experience being 

I 20 ignored or only selectively applied in the haste to get decisions through. 
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• Lack of incentives to learn - unless there are proper accountability loops 
built into the project cycle there may be little incentive to learn. This is par- 
ticularly the case when staff or consultants shift from task to task, and have 
generally moved on long before the consequences of failure to learn are felt. 

• Tunnel vision - the tendency of some staff or operational units to get 
stuck in a rut, carrying on with what they know, even when the shortcom- 
ings of the old familiar approaches are widely accepted. 

• Loss of institutional memory - caused by frequent staff rotation or heavy 
reliance on short-term consultants, or by the weakening or disbanding of 
specialist departments. 

• Insecurity and the pace of change - if staff are insecure or unclear what 
their objectives are, or if the departmental priorities are frequently shifting, 
this can have an adverse effect on learning. 

• The unequal nature of the aid relationship - which tends to put donors in 
the driving seat, thereby inhibiting real partnerships and two-way knowledge 
sharing. 

Many of these obstacles are deep seated, and cannot be changed overnight. 
However there was a broad acceptance of the need to "dig deep” to locate where 
the obstacles to learning lie, and to find ways they can be attacked. Agencies 
expressed this in different ways, but many recognise that this is not just a ques- 
tion of fine-tuning; what is needed is a radical rethinking of how organisations 
learn and are held accountable. 

3.4. Addressing the demand side 

A common feature of most evaluation feedback is that it tends to be supply 
driven, with the evaluation unit pumping out information on the assumption that it 
will be picked up. Though inevitable to some degree, several agencies pinpointed 
this as a key weakness in current feedback practices. Evaluation units were doing 
a reasonably good job on the supply side, as CIDA put it; it is the demand side 
that is their " Achilles heel". 1 

The introduction of evaluation "help desks” is one concrete way of addressing 
this. This idea has been tried out at the World Bank and is seen as a successful 
way of pointing people to relevant evaluation lessons (both within and outside 
the Bank) as and when they need the information. Help desk queries are currently 
running at I 000 per year, with 60% coming from outside the organisation. The 
Netherlands also plans to set up a help desk or knowledge centre to support 
decentralised evaluations. 


1 . Making this point another way, FAO commented that since demands are changing all the 

time, evaluation units need to "respond to these changes if they are to survive". 21 | 
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4. Understanding Target Audiences 


4.1 . Better targeting 

The Synthesis Report made a start at separating out some of the main audi- 
ences for evaluation feedback, distinguishing twelve main target groups and indi- 
cating which are likely to be priorities for learning purposes, as opposed to 
accountability. The aim was to underline both the diversity of audiences that evalua- 
tion units may be aiming to reach, the different reasons for wishing to target them, and 
the very basic point that one feedback approach is not going to suit them all. 

This was a first approximation, and the Workshop provided an opportunity to 
break down target audiences further and start to look in more detail at different 
sub-groups. 1 No attempt was made to draw up a comprehensive list of all possible 
audiences for evaluation feedback. The essential point is that priority audiences 
vary depending on the type of evaluation, the messages coming out, and the par- 
ticular context in which evaluations are taking place. There was a recognition that 
more effort is needed in identifying and prioritising target audiences in specific 
cases, and that attention needs to be given to this early on in the process, not 
after the evaluation is completed. 

Careful targeting allows intelligent tailoring of feedback approaches, a point that 
was underscored in the presentation by DANIDA. This broke down external audiences 
into three main groups and explained how different feedback "vehicles’’ are used to 
target each of them (see Box 2). The point was made that given the diversity of poten- 
tial audiences it is not realistic to try and reach everyone; agencies have to decide 
who is most important in specific cases, and make sure they get to them effectively. 

A number of audience groups were discussed in more detail. Experience of 
agencies in relation to four of the important external audiences within donor countries 
is summarised below. 

4.2. Parliament, ministers and boards 

For bilateral agencies, key ministers and parliamentary committees are often 
a crucial target audience, whereas governing boards of various forms play the 
I 22 equivalent role for most multilateral agencies. In both cases they require delicate 
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Box 2. Danida’s approach in matching feedback vehicles 
to specific audiences 




AUDIENCE GROUPS 


Primary purpose 

Accountability 

Learning 

Partners 

FEEDBACK 

VEHICLES 

Academics, students, 
Parliament researchers 

Opinion makers/leaders External resource base 
General public (consultants, etc.) 

NGOs 

Developing country 
partners 

Other development 
agencies 

Evaluation reports 


• 

• 

4 page summary 
25 page popular version 

• 

• 

• 

• 

Press events 
Video/film 

Annual report to Board 
Danida’s annual report 
Danid@visen newsletter 
Public meetings and 

• 

• 

• 

• 

• 


• 

professional associations 


• 


Lectures at universities 




and high schools 


• 


World Wide Web 

• 

• 

• 

Seminars/workshops 



• 

Participation in evaluations 



• 

Source. Danida (2000), "External 

Feedback - Danida's 

Dissemination Vehicles", 

paper presented to DAC 


Tokyo Workshop on Evaluation Feedback for Effective Learning and Accountability, September 2000. 


handling. If good relationships are created, based on trust and regular dialogue, 
they can be a vital ally; anything less can spell serious problems for the agency. 

Specific situations vary a great deal, depending on the reporting structures 
and personalities involved. For example: 

• In Norway, aid evaluation is located in the Policy Planning and Evaluation 
Department of the Ministry of Foreign Affairs. When an evaluation report 
is finalised, a memorandum containing proposals for decisions is drafted 
and presented to the Minister and relevant political staff in order for follow- 
up decisions to be made. These meetings are a recent development and 
have provided a useful interface between the operational and the political 23 I 
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levels, helping to raise awareness among political staff of the dilemmas 
and systemic difficulties in development cooperation. They have put the 
onus on the evaluation unit to make sure that recommendations are concrete 
and well thought out, and are based on a consensus among the relevant 
parties involved (a process which can take time, but which is valuable in 
its own right, and essential in terms of subsequent impact]. 

• In the Netherlands, there is an established system for feedback to Parlia- 
ment. Evaluation reports are presented to the relevant Minister who then 
passes them on with an accompanying "policy reaction” for discussion in 
Parliament. Creating a routine of regular meetings has helped sustain 
interest among the political staff, and means that bad news as well as 
good news can be discussed in a more balanced and informed way. 

• In the UK, the Parliamentary Committee on International Development 
established by the current government has created a new axis of influence 
and power, and is keeping a much closer eye on the performance of DFID 
than in the past. It receives submissions from a wide range of sources and 
provides a very public forum where policy issues and controversies are aired. 

A general message that came through at the Workshop is that frankness and 
openness are essential components in developing good relationships with these 
kinds of high level audiences. USAID commented how the US Congress are partic- 
ularly suspicious of "glowing reports", and has appreciated the Agency's efforts to 
be honest about where difficulties have occurred. 

4.3. The Media 

This same point was also emphasised in relation to feedback to the media, 
although relatively few evaluation units have a lot of experience working with the 
media. The risk of evaluation material being selectively quoted or used to create 
potentially damaging headlines was acknowledged. But, at the same time, most 
recognise the important role of the media in holding agencies accountability and 
the valuable part they can play in raising awareness about development assis- 
tance more generally . 2 

Senior management are highly sensitive to media reports - both positive and 
negative - so getting media relations right is important. DANIDA is one the agen- 
cies that has done the most in terms of working with the media. They made the 
point that damaging headlines are based on three ingredients: a) a development 
fiasco of some kind, that b ) is being covered up, and c) is an exclusive story. By adopt- 
ing an open policy, and releasing evaluation reports widely, the last two of these ingre- 
dients can be removed. So press releases are routinely issued, and these are backed 
up by press conferences which provide an opportunity for in depth explanation of the 
I 24 evaluation results. This approach has not solved the problem of occasional 
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development fiascos, but it has led to a much more mature and constructive relation- 
ship with the media, which in turn has been important in maintaining the widespread 
support for development assistance among the Danish public. 3 

4.4. NGOs and CSOs 

Non-governmental organisations (NGOs) and civil society organisations (CSOs) of 
various kinds are seen as an important audience for evaluation feedback, as well as 
partners in the evaluation process, and sometimes pioneers in developing new evalu- 
ation approaches, such as participatory monitoring and evaluation. Some agencies 
have taken steps to engage with these groups more systematically. The French now 
have a High Council for international Cooperation which includes academics and 
NGOs and enables an exchange of opinion with the government about development 
issues. Some agencies have a strategy of sending reports to key NGOs, CSOs, aca- 
demics and other leading opinion makers in the development arena. 

Establishing a constructive relationship with such groups is seen as an impor- 
tant step in broadening out the learning loop, and demonstrating accountability. But 
this is not without its pitfalls. One agency spoke of its experience in bringing NGOs in 
as participants in a major sectoral review, and how it was taken off guard when, the day 
the before the publication of a major report, some of the NGOs released a press state- 
ment attacking the document. From this they have learned that in future consultations 
of this kind they need to establish clear "ground rules” and be alert to the possible 
diverging agendas involved, particularly when working with NGOs that have an active 
campaigning role and a sophisticated media department to go with it. 

4.5. The general public 

The general public within donor countries are not normally regarded as an 
audience that can be reached directly with evaluation feedback. However, )apan 
underlined that the budget for ODA is provided by the general public as taxpay- 
ers, and that therefore the public is important as a target audience, lapan has pio- 
neered a programme to involve "ordinary people’ as citizens" monitors of the 
country’s development assistance effort. 

The scheme started in 1999 and is deliberately targeted at non-specialists 
such as "small shop owners from down the street", "students", "fishermen", or 
"ordinary housewives". Groups are made up of eight to ten volunteers. They start 
with a one-day orientation seminar, then spend a week in the field visiting 
projects in several countries. They are asked to write a short report on their return 
giving their common sense assessment of what they saw. These are combined into 
one document, so as to present a range of impressions side-by-side. These are 
then distributed in large numbers by the Ministry of Foreign Affairs, using public 
outlets such as libraries, schools and the Internet. 25 I 
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The scheme has proved popular and has been steadily expanding. About 
I 00 separate monitoring trips were organised in the scheme this year. The scheme 
does not claim to be rigorous, and it is not appropriate for all kinds of development 
projects. 4 But it has succeeded in shedding a totally new light on the Japanese aid 
effort, one which seems to be appreciated by the "ordinary people” whose taxes 
fund the aid programme. 

4.6. The case for disclosure 

There was a broad consensus on the desirability of full disclosure of evalua- 
tion reports and considerable progress has been made in this direction in recent 
years. Several agencies admitted, however, that theory and practice were not 
always the same thing, and that they needed to do more ensure that reports are 
made available more promptly and systematically. 

Disclosure of negative or controversial evaluation findings can obviously create 
difficulties for agencies, so it is important not to be naive about possible fallout. But 
the overall view was that the benefits of disclosure in the long run make it worthwhile. 
Agency credibility is enhanced, and the willingness to publish negative reports helps 
to boost the validity of the positive ones. 5 Greater disclosure can also increase the 
pressure for more systematic follow-up of recommendations, while motivating those 
involved in evaluations to produce a better product, since they know their report will 
be made public, rather than being buried on a shelf somewhere. 


Notes 


L2L 


1 . The wide range of potential audiences for evaluation feedback was underlined in the pre- 
sentation by Germany, which identified 17 main actors in the aid sector, within Germany 
alone. 

2. |apan has taken this a step further and has included journalists in evaluation teams. 

3. It was noted that according to a recent opinion survey in Denmark there is 73% support for 
the 1% GNP target for development assistance, even though only 50% believe that aid 
works. 

4. Practical projects such road building schemes, or support for elementary schools, are 
easiest for non-specialists to assess. This approach would not be appropriate for more 
complex projects such as balance of payments support. 

5. One agency commented that in making all its evaluation reports public it recognises 
that it is providing potential ammunition for critics. But it wants to develop a reputation 
as being an agency "that can make mistakes 1 '. 
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5. Getting the Message Across 


The Workshop provided a valuable opportunity for agencies to share experi- 
ence on feedback approaches. Some of the most important communications lessons 
that emerged are summarised here. 

5.1. Learning by doing 

It was agreed that "learning by doing" is often the most effective feedback 
route. People who are directly involved in evaluations, or the consultations sur- 
rounding them, have the greatest exposure to the specific lessons being learned 
and the context in which they apply. They are in the best position to internalise 
these lessons, and if the activity is to continue, to feed them in to the next stage of 
planning and implementation. It therefore makes sense to maximise these oppor- 
tunities where possible, broadening out the consultation loop, and taking the time 
needed to share drafts and encourage buy-in to the evaluation process among key 
stakeholders. DANIDA and IFAD are among the agencies that placed particular 
emphasis on the need to do more in this area. 

Though clearly important, this kind of feedback is obviously not the end of 
the story. Lessons need to be spread much more broadly if they are to have an 
impact beyond the immediate stakeholder group. This means sharing lessons with 
a wide variety of other audiences. The problem is that most of these are bom- 
barded with material from other sources, are already suffering from information 
overload, and may have little time or motivation to pick up on lessons learned 
from evaluations that they have no direct connection with and no responsibility 
for. So this is where some of the biggest communication challenges lie. 

5.2. Relevance 

The perceived relevance of evaluation lessons is of prime importance in 
determining whether they will be picked up. The classic criticism of project evalu- 
ations is that the lessons emerging are either too specific or too general. An 
important way that agencies are addressing this is by shifting emphasis towards 
higher level, country-based, or thematic evaluations. These are likely to have 
much wider relevance and are therefore worth investing more in. 27 I 
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In some agencies there is an active debate on whether it is still appropriate 
for central evaluation units to carry out individual project evaluations at all. Some 
argue that this should be devolved entirely to operational departments, allowing 
evaluation units to concentrate on thematic or country work. Others make the case 
that independent project-level evaluations are an essential raw material for higher 
level analyses, so need to be continued in parallel. 

5.3. Timeliness 

Closely related is the question of timeliness. Because of the lag time in pro- 
ducing final reports and other spin-off products, evaluation feedback often ends 
up being "too much, too late", as the Inter-American Development Bank put it. 
This can mean that the substantial investment in producing evaluation outputs is 
largely wasted, since the messages contained are written off as being out-of-date. 
This is particularly the case with ex-post evaluations, which tend to be looking 
back at initiatives that were planned quite a long time ago, and therefore reflect 
previous generations of aid thinking. Though the lessons emerging may be valid, 
that will often be dismissed as being no longer relevant. 

Various approaches were put forward as ways of addressing this: 

• Planning ahead so that evaluation outputs are timed to coincide with major 
policy events, for example, key conferences, or the preparation of sector or 
country strategies. These provide an ideal opportunity for evaluation lessons 
to be fed into broader processes. 

• Putting more emphasis on mid-term evaluations, so the results can feed 
directly into subsequent planning. 

• Finding ways of encouraging feedback during the evaluation process. There 
is obviously a balance here between obtaining final sign-off from all the 
stakeholders involved, and getting results out quickly. But if this can be 
tipped in favour of early release, especially when results are particularly signif- 
icant or controversial, this is likely to increase the chance of messages 
being picked up. A briefing note to senior management in advance of the 
publication of evaluation reports, for example, can be a useful way of alert- 
ing them to evaluation findings while they are still "fresh". This can increase 
the possibility of early follow-up action, while giving managers a chance to pre- 
pare an appropriate response if major problems are uncovered. 

5.4. Quality issues 

Several agencies stressed the importance of rigour and precision in drawing 
up evaluation lessons and good practice guidelines. Where recommendations are 
I 28 too vague or diffuse, where the evidence and analysis backing them up is weak, 
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and where no distinctions are made between lessons that are locally applicable 
and those that have wider relevance, it is hardly surprising that messages fail to 
get through. Indeed, evaluation reports that are seen as sub-standard can do long 
term damage to the credibility of the evaluation unit. 

A variety of approaches can be used to help assure quality - for example, ref- 
erence groups, and peer review processes. Well-designed iterative consultation 
processes also help to raise standards. The point was made that evaluation units 
"live or die by the quality of their work", and that is a fundamental issue that need to be 
addressed. Some are tackling this specifically. The World Bank, for instance, is 
currently developing guidelines and training to improve the quality of evaluation 
recommendations. 

There were differing views, however, about the extent to which evaluation les- 
sons can be successfully codified into good practice guidelines. EBRD sees this as 
a key function for evaluation units, whereas some other agencies argued that with 
the types of development assistance they were providing, the specificity of local 
country situations makes it difficult (and potentially dangerous) to generalise in 
this way. 

5.5. Language barriers 

The way that messages are phrased, and the language they are in, are crucial 
in terms of impact - whatever the medium being used. The Workshop under- 
scored the importance of: 

• Brevity - messages need to be presented is a way that is short, arresting, 
and to the point, otherwise many audiences will simply pass them by. "Wc 
need to recognise that the time of people we're trying to reach is very valuable", as one 
participant put it. Yet all too often the impression is that consultant's 
reports are paid for by the kilo, rather than on the basis of their usefulness 
or digestibility. The more senior the audience, the more succinct the mes- 
sage needs to be. But in presenting "the tip of the iceberg", it was stressed 
that the rest of the iceberg needs to be there to support it. 

• Terminology and language - cutting out technical jargon is the first step. 

But most agencies are also acutely aware of the huge communication gap 
that exists when evaluation results are only made available in one lan- 
guage (usually English). This does not imply that all reports need to be 
translated into local languages, or that donor agencies are necessarily the 
best placed bodies to do this (in country partners may be in a better posi- 
tion to take it on). However, the need to address this major failing - possi- 
bly in creative ways using radio, TV or video, for example, rather than 
paper-based approaches - was repeatedly underscored. 29 I 
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5.6. Intelligent use of the Internet 

The new generation of internet-based communication approaches, including 
websites and internal intranets, are widely seen as a valuable addition to the evalua- 
tion toolkit. From a disclosure perspective, publishing evaluation reports on the 
agency's public website is an excellent and cost-effective way of opening up access to 
the material, although not necessarily a very targeted one. In addition, the Internet 
has proved to be effective not only for disclosure but also as a database. It can also be 
a useful consultation tool. In the recent World Bank review of the forestry sector, for 
example, the draft placed on the website attracted comments from over 200 people. 

Internet tools need to be used well, however, in order to reap these benefits. 
While Working Group members have made an important start in setting up evalua- 
tion sections on their agency websites and/or intranets, most recognised they have a 
considerable way to go in climbing this new learning curve. A number commented 
that their "first generation" websites and intranets were due for an overhaul, and 
that further work was needed to make them more accessible, up-to-date, and easy 
to navigate. 

As more and more evaluation reports are made available electronically, the 
question emerges of how this data can be intelligently searched in order to extract 
the most relevant material when you type in search terms, rather than being 
swamped with screens full of variable quality data matches that need to be waded 
through. One way of tackling this problem is though more thorough indexing and 
"tagging” of material, so the contents of documents are more clearly flagged with 
keywords. Another is though using new data "mining" tools. The World Bank has 
been investigating the potential for using computer-based linguistic analysis. 
They have tested this out on a subset of I 200 evaluation summaries, with encour- 
aging early results, suggesting that these tools can helpful in zeroing in on relevant 
material and unearthing “what we don't know we know". 

Another creative use of technology was mentioned by CIDA, which is devel- 
oping a computer-based system whereby relevant prompting points "pop up” on 
staff computer screens at key stages in the project development and approval 
process. These link in to ClDA’s lessons learned database, and are a reminder to 
staff to consult previous experience. 

But the limits of new technology and in particular the Internet were also 
accepted. Though connectivity is improving fast, there are still big problems with 
Internet access for many developing country audiences. There is also a strong ten- 
dency for systems to be designed based around supply side criteria, rather than a 
clear understanding of the way information is actually used. So it was recognised 
that while computer-based systems can be highly effective and need to be 
exploited, they are not a panacea, or a substitute for conventional feedback 
I 30 approaches. 
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5.7. Interagency lesson learning 

The need for more effective sharing of evaluation lessons between donor 
agencies has long been recognised. This need is becoming even more acute with 
the advent of the new generation of Sector Wide Approaches and the introduction 
of Poverty Reduction Strategies. These and other joint initiatives' are premised on 
the assumption that coordinated agency action will be more effective than indi- 
vidual efforts. Yet mechanisms for exchanging evaluation lessons between agen- 
cies are still weak, and practical hurdles continue to get in the way of more 
frequent joint evaluations - which, when they do occur, are generally seen as a 
very good way of sharing lessons and methodologies. It was suggested that agen- 
cies have “not been very imaginative” in tackling this problem. 

The DAC Inventory of evaluation summaries, set up in 1988, was an important 
first step in improving lesson sharing. 2 However, comments at the Workshop con- 
firmed the view that the potential for the Inventory has yet to be fully realised and 
that it suffers from a number of significant shortcomings - notably its patchy cover- 
age due to the inconsistent submission of data by many donors, the prevalence of 
older material in the database, its limited functionality, and that fact that it is not 
systematically cleaned. 

The Workshop agreed that there is a need for a fresh look at the Inventory, 
and that this might be usefully discussed at the next Working Party meeting. With 
the increased publication of evaluation reports on donors' own websites, it was 
suggested that a more decentralised approach to data storage might be prefera- 
ble, with a user-friendly central search engine pointing users to relevant reports on 
agency websites. This would eliminate the problem of donors having to remember to 
submit summaries to the central database, though there would still be issues to tackle 
due to the lack of a standard format and approach in preparing reports. 

One possibility would be for the redesign of this facility to be incorporated as 
part of the Global Development Gateway initiative being led by the World Bank. 3 
The DAC is already collaborating with this in carrying out tests on distributed data 
sharing. There is some debate, however, about the scope and direction of this ini- 
tiative, so views may differ about the wisdom of linking these together. 

5.8. The value of face-to-face communication 

in evaluation feedback, as in other spheres, face-to-face dialogue is still the 
most effective way of sharing ideas and getting messages across. While recognis- 
ing that new communication approaches are opening up important new channels 
of influence, several agencies made a point of underlining the continuing value of 
meetings, seminars and briefings of various kinds - both within the organisation 
and externally - as a indispensable focus for learning. 
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Notes 


1. For example, New Zealand is preparing recommendations on co-financing policies with 
multi-laterals and USAID has recently agreed to undertake pooled funding programmes. 

2. The DAC inventory is hosted by CIDA and IDRC. See mimweb.idrc.ca/cida/dadoge.ktm. 

3. See www.worldbank.org/gateway. 
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6. Involving Partners 


There was widespread recognition at the Workshop of the need to do far more 
to involve partner country stakeholders - both as participants in the evaluation 
process and as audiences for the lessons that emerge. Although agencies were 
able to point to examples of where efforts had been made, most were very frank 
about their shortcomings in this area. SIDA is one of the few that that have gone as 
far as to look at how big this gap really is, and where the problems lie. 

Reporting on a detailed study of the impact of nine previous evaluations 
among different groups of stakeholders , 1 SIDA presented a revealing picture of 
how evaluation lessons fail to reach many of them, particularly those in partner 
countries. The report concluded that: "The further away you are from the centre of the 
project in terms of power and involvement, the less information you receive on evaluation recom- 
mendations and findings." Many stakeholders were barred from acting on the results 
of evaluations because they never get to see the evaluation findings . 2 

The study also looked at the political context in which evaluations are conducted, 
noting that power relations tend to be heavily weighted in favour of the donor. It 
went on to state that "for the majority of stakeholder the evaluation process could just as well 
have been left undone" and that "in terms of adherence to the present policy framework of 
Swedish aid ( which stress partnership and ownership), the evaluation process as we know it is in 
dire need of reform". 

These hard-hitting conclusions underlined the size of the challenge facing 
agencies if they are to turn the rhetoric of partnership into a reality . 3 Among the 
points raised in discussion were the following: 

• Starting early - trying to build meaningful participation in to ex-post eval- 
uations is difficult if the project has not been planned or implemented in a 
participatory way. Buy-in from partners is likely to be much greater if they 
are involved right from the beginning of the project cycle. 

• Capacity building - is an important priority if in-country partners are to be 
more fully involved in evaluations, and is a particular challenge in countries 
that are implementing active decentralisation policies. A number of initia- 
tives were mentioned in this context . 4 
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• Trade-offs - although involving partners in evaluation work is a highly 
desirable goal, the process takes longer and there may be compromises to 
be negotiated in terms of maintaining rigid standards of quality and impar- 
tiality. But these are trade-offs worth accepting if the process leads to 
greater buy-in and learning. 

• Risks - in cases where controversial or politically unpopular findings are 
emerging, there may be risks for the local nationals involved in evaluation 
teams, either at a personal or organisational level. Evaluation teams need 
to be sensitive to these risks otherwise they can leave local team members 
in a very awkward situation when the visiting team has gone. 

• Seminars - in-country seminars are a valuable way of encouraging greater 
participation. Difficult messages can sometimes be delivered more easily if 
they are presented orally, especially in countries where there a culture of 
consensus being reached through discussion, rather than through reports 
on paper. 

• Participatory monitoring and evaluation (PM&E) - was recognised as a 
valuable addition to the evaluation toolkit, and one which agencies need 
to gain experience in and set aside adequate time for. Engaging partner 
countries in evaluation feedback systems is essential. The need for capac- 
ity building in partner countries is crucial. There was wide consensus on the 
need for integrating evaluation feedback into the planning and monitoring 
of programs and strategies in partner countries. The need for establishing 
and collecting the necessary indicators was also underlined, lapan sug- 
gested that similar workshops should be held involving development 
agencies and stakeholders in partner countries as well as development 
agency evaluators. But some stressed that this needs to be kept in per- 
spective, as it is not the answer in all situations. 

An encouraging picture of how coordination between donors and partner 
countries can work was presented by the representative from the Lao PDR. She 
described a new Roundtable Process that has been introduced involving annual 
meetings between donors and government representatives . 5 These are designed 
to harmonise donor funding with evolving national development priorities, and to 
“improve national development planning, programming , monitoring and evaluation, by continuously 
integrating lessons learned through an effective evaluation feedback mechanism". 

This has opened up a useful “two-way street" for sharing lessons and priorities 
and has provided a forum where pressure can be applied on donors for them to 
be more accountable and to coordinate their efforts better, including sharing eval- 
uation results. The point was also made that lesson learning needs to happen as 
soon as possible in the project cycle. This implies the need to share project moni- 
I 34 toring information, not wait for ex-post evaluations before lessons can be learned. 
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Box 3. Examples of partner country involvement: The Netherlands' 

Trying to improve feedback in the partner country: the case of country 
programme evaluations 

IOB has undertaken seven country programme evaluations, in Bangladesh, 
Bolivia, Egypt, India, Mali, Tanzania and the Palestinian Territories. The overall 
objective of each of these studies was to assess the policy relevance, effective- 
ness, efficiency and sustainability of the results of the bilateral aid programme. 
Each one covered the full period of the bilateral development cooperation rela- 
tionship and had a similar approach and structure. The evaluation started with an 
inventory of all activities. Subsequently, a number of sectors were selected for 
detailed evaluation involving field investigations. Teams of independent experts, 
including Dutch and local evaluators, carried out all studies. Each country pro- 
gramme evaluation was coordinated by an IOB staff member and one external 
advisor, who were responsible for writing the final report. 2 

Attempts were made to involve the authorities, other stakeholders and local experts 
in the design and implementation of the country programme evaluations in orderto: 

• improve the quality of the evaluation by incorporating the views of these 
institutions into the analysis and making use of their detailed knowledge of 
society and social change in the partner country; and 

• stimulate the feedback of the evaluation in the partner country by exchang- 
ing ideas about the results of the evaluation and, where possible, reaching 
a shared understanding of the issues to be addressed in orderto improve 
policy implementation. 

Participation took place through a combination of approaches, which generally 
involved sharing draft terms of reference with the local authorities, conducting inten- 
sive discussions with partner institutions at various stages of the evaluation, working 
with an in-country advisory group of independent experts, involving local experts as 
evaluators, organising workshops with stakeholders to discuss the setup of field stud- 
ies, sharing preliminary results and organising a seminar involving all relevant stake- 
holders at the end of the evaluation study. Moreover, the official response by the 
partner country authorities was incorporated into the final evaluation report. 

Feedback involving in-country stakeholders: the study "Institutional 
Development in the Water Sector", Mozambique 3 

Following field investigations and initial analysis of the results of the study, 
IOB organised a seminar in Maputo in order to share the results with a wide range 
of stakeholders such as the embassy, the field staff of the development activities 
studied, senior staff of counterpart institutions, and other donors. During the 
meeting the purpose of the evaluation was reiterated and its results presented. 
The participants were invited to voice their views on the preliminary outcomes of 
the study. Where points of view differed and where the accuracy of the findings 
had been questioned, the evaluators followed up with additional investigations 
and analyses. Later, the draft of the evaluation report was distributed among the 
stakeholders, who were invited to comment on its contents. 
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Box 3. Examples of partner country involvement: 

The Netherlands ( cont .) 

Sharing evaluation information with stakeholders and encouraging them to 
participate at an early stage contributed to the acceptance and implementation of 
the findings. It also gave donors sufficient time to adapt their planned activities in 
the water sector. Counterpart organisations were able to use the information in 
negotiations with the Netherlands and other donors, which were scheduled to 
take place before the final report on the IOB evaluation could be published. 


1. From "Evaluation Feedback: Experiences and Recent Changes”, R. D. van den Berg and 
T. Kliest, paper prepared for the DAC Working Party on Aid Evaluation Workshop, 
Tokyo, 26-28 September 2000. 

2. For a full description of the way in which lOB’s country programme evaluation on Egypt 
was conducted, refer to the proceedings of the DAC Working Party on Aid Evaluation’s 
seminar "Evaluating Country Programmes”, Evaluation and Aid Effectiveness No. 2, 
OECD/DAC, 1999. 

3. Institutional Development: Netherlands Support to the Water Sector, 1988-1998. IOB 
Evaluations No. 284, March 2000. 


Box 4. Partner country perspectives: some highlights 

Three partner country representatives at the workshop (Indonesia, Laos, and 
Philippines) presented their respective systems for dealing with evaluation feed- 
back, and current efforts to improve them. 

Indonesia 

Bappenas (the National Development Planning Agency) issued a Ministerial 
Decree in 1996 offering guidelines to line agencies and institutions involved in 
development planning to implement a Performance Evaluation System. These 
guidelines emphasise results, outcomes and impact and provide information on 
how to set project priorities based on these elements. The most recent decree 
(2000) further elaborates on evaluation, providing illustrative examples for further 
clarification. A separate set of guidelines for monitoring systems has been devel- 
oped. Within this evolving system of evaluation and monitoring, emphasis is 
being put on disseminating information from evaluation studies to relevant audi- 
ences. Although a large part of evaluation activities is de-centralised, undertaken by 
line agencies and provincial and district government, effort is made to ensure the 
feedback/dissemination of results "upwards" to central policy makers and planners. 
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Box 4. Partner country perspectives: some highlights (font.) 

Laos PDR 

The government of Laos has launched a new "dialogue process" with their 
partners in development. The intent is to change the traditional Round Table 
Meetings into a dynamic process instead of periodic meetings. This new approach 
sets three main objectives. Among these is the "improved national development 
planning, programming, monitoring and evaluation, by continuously integrating 
the lessons learned through an effective evaluation feedback mechanism...”. The 
government is currently in the process of "institutionalising” the process of evalua- 
tion feedback in order to further integrate results into policy. 

Laos is also in the process of an extensive decentralisation effort, whereby prov- 
inces are being given new responsibilities. Implementing units are required to use 
evaluation systems and monitor their development plans. It is expected that decen- 
tralisation will make it easier to translate evaluation feedback into improved ongoing 
projects and future programmes due to closer links between feedback levels. 

Philippines 

In the Philippines implementing agencies are required to report on project out- 
comes and impact in the drive towards ensuring that the objectives of development 
projects are achieved (National Economic and Development Authority (NEDA) 
Board Resolution No. 3 of 1999). The Medium-Term Philippine Development 
Plan (1 999-2004) provides for the setting up of a public expenditure management 
system, which includes the establishment of a system to enhance data collection and 
reporting of results and outputs of various agency programmes and activities. Corre- 
sponding to this mandate, implementing agencies are required to incorporate Results 
Monitoring and Evaluation (RME) in project plans and proposals. Information 
obtained through evaluations will be submitted to NEDA, and are then integrated into 
the Annual ODA Portfolio Review, which in turn is used as a tool in planning future 
activities. 
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Notes 


1 . "Are Evaluations Useful? Cases from Swedish Development Cooperation", ). Carlsson et at., 
Sida Studies in Evaluation 99/1. 

2. The study distinguished between different kinds of use of evaluation results, contrast- 
ing "instrumental use" (where results feed directly into some kind of action), with "con- 
ceptual use" (where they feed into policy processes), "legitimising use” (where they are 
used to justify existing positions, "ritual use" Iwhere they are largely symbolic), and "no 
use” (which speaks for itself). Worryingly, it found that "instrumental use” is quite rare 
and that the most common pattern was "no use”. 

3. IFAD talked of the need to consider "shifting the fulcrum of evaluation feedback to the 
South". 

4. For example, the Asian Development Bank is sponsoring a major programme in China 
to develop capacity for budget monitoring at a decentralised level; several representa- 
tives from this programme participated in the Workshop. The World Bank is developing 
an "International Program for Development Evaluation Training" (IPDET) in partnership 
with several other institutions. 

5. The process is coordinated by the Committee for Investment and Cooperation, in the 
Prime Minister’s Office. 
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7. Institutional Issues 


7. 1 . The status of evaluation units 

Underlying many of the Workshop debates was the question of the status of 
evaluation units within agencies. There is no doubt that this is a big factor in 
determining their impact. Evaluation units that are seen as being marginalised 
and remote from the centre of power are hardly likely to have a major influence in 
organisational learning terms, even if they are doing good work. To be taken seri- 
ously, evaluation units need to have strong senior management backing, with staff 
of sufficient status and qualifications. Some agencies argued that it is particularly 
important to have staff with experience at the implementation level. This is one 
way of avoiding the "ivory tower” tendency - either real or perceived. 

There was hot debate over the pros and cons of independence. Some agen- 
cies said that it is fundamental to the role of evaluation units, and is in fact their 
key attribute. Others argued that it can be a hindrance if independence becomes 
a barrier to partnership and dialogue. For example, FAO suggested that in a learn- 
ing organisation with teamwork and a flow of knowledge within a learning cycle it 
may be more appropriate to de-emphasise the need for full independence, as 
there may be a risk that “independence can lead to irrelevance". 

There were similar divergences on the merits of rotating staff between evaluation 
units and operations. Some agencies advocate stability, which allows knowledge and 
expertise to accumulate, and makes it easier to maintain an independent role; the 
contrary view was that staff rotation is beneficial and encourages learning throughout 
the agency. However staffing is organised, it is was pointed out that an "us and them” 
view of operations and evaluation is not the best mindset for a learning organisation, 
and that where staff rotation does occur, it is helpful to have a system of debriefing to 
make sure experience is passed on rather than being dispersed. 

Where there is a legacy of evaluation units being seen as policemen there is 
clearly work to be done. In the past, evaluation and audit functions tended to be 
perceived as being mostly about highlighting mistakes and allocating blame. With 
the shift to a more explicit learning role, it is possible to move on from this, although 
this transition will take time. Inevitably, evaluators have to be bearers of bad news at 
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times, pointing out what has not worked. They need to balance this with efforts to 
communicate what the good news is, and where successes have occurred. 

7.2. Institutionalising lesson learning 

Institutionalising lesson learning, so that it becomes more reliable and sys- 
tematic, was widely seen as being a key priority. "Learning cannot be optional", as one 
participant put it. 

A variety of examples were presented of where specific steps had been intro- 
duced to encourage this. These included the European Commission’s use of a 
“Fiche Contradictoire" , a two-column table with the recommendations drawn out by 
the Evaluation team in the left hand column, and the responses and action taken 
by those responsible in the right hand column. This is made available on the 
intranet for comment and published with the evaluation report. 

The challenge is to make such systems effective in practice, not just on paper. 
Some of the points raised in this regard were: 

• Leadership - senior management needs to champion lesson learning and 
recognise that it may mean working in different ways across the agency, 
including at senior management level. This is a point that is well accepted 
in the corporate sector. Quoting one leading source: "The quality of leadership, 
embodied in the top management team process, determines the organisation's ability to 
learn ." 1 It was suggested by Japan that senior management would benefit from 
specific seminars to underline this message and make clear their pivotal role. 

• Consultants - the high level of contracting out of work is an important factor 
to be considered. Consultants should be a key group in a learning organi- 
sation - both as producers and users of knowledge - but they are rarely 
identified or treated as such. 

• Staffing issues - high staff turnover and staff shortages are a significant 
barrier to institutionalising lesson learning. 

• Carrot and stick - improved packaging and distribution of evaluation lessons 
will not automatically lead to institutionalising lesson learning. Incentive 
structures to encourage learning need to be built into the system, or else 
penalties established for not taking account of previous lessons. 

7.3. Results based management 

As the Synthesis Report pointed out, there are parallels between evaluation 
feedback and some of the concepts involved in Results Based Management 


1. From "How Organisations Learn", edited by Ken Starkey, International Thomson Business 
40 Press, 1996. 
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(RBM). The Workshop highlighted the varying levels of enthusiasm for RBM among 
agencies, as well as the different ways it is used. 

USAID introduced Results Based Management in 1994 in order to encourage 
decentralisation of decision-making, and link budget allocations to objectives 
rather than activities. They presented a paper describing their experience. Some 
of the lessons they highlighted included the importance of collaborating with part- 
ners in performance management, and the need to use a variety of performance 
indicators and balance annual performance assessment and reporting with longer- 
term sustainable issues. They also stressed that assessment of results is just one 
factor in resource allocation decisions; other important factors include develop- 
ment needs, foreign policy considerations, and contributions of other donors. 

A number of other variants on Results Based Management were also dis- 
cussed. The DFID presentation focussed on the way the agency is responding to 
the introduction of "Public Service Agreements", a government-wide initiative set- 
ting down commitments and targets over a three-year period, in the case of DFID, 
these are tied to the International Development Targets. Their presentation pointed 
out the strengths and weaknesses of the approach, but concluded that it does provide 
opportunities for linking decisions to performance more intelligently, provided it is 
applied carefully in a flexible, not mechanical, way. A paper was also submitted by 
Indonesia on their experience with performance evaluation as a management tool. 

Evaluation units can play an important role as advocates of RBM approach, 
although there were mixed views as to how active this role can be. Some made 
that point that RBM is more useful for operational or policy departments, rather 
than evaluation work. 

7.4. Staffing and resourcing 

The question was posed as to whether improving evaluation feedback is 
going to be constrained by lack of resources, since many of the measures dis- 
cussed will undoubtedly cost money. The consensus was that this is generally not 
the case. The constraints that exist are more to do with lack of time, and not hav- 
ing the right blend of staff skills. Provided the case for better evaluation feedback 
is clearly stated, and it is treated as a sufficient priority, most agencies felt that 
resources could be made available. And where there are trade-offs that have to be 
made, the clear view was that it is better to do fewer evaluations, but make sure 
the lessons from them are properly communicated, rather than carrying on with a 
large evaluation portfolio and wasting the opportunities they provide for learning. 
Otherwise, “what is the point?" as the Netherlands put it. 

New staff skills will certainly be needed if agencies are going to gear up their 
feedback efforts in a significant way. Particularly important will be communication 
skills - in areas such as editing, translating, web site management, facilitation of 
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LiL 


Box 5. Japan’s efforts to strengthen feedback and evaluation* 

Evaluation of development co-operation activities has recently received 
increased attention from the Japanese public. Budgetary constraints and public 
concern about ODA effectiveness has prompted a demand for greater transpar- 
ency and accountability. Consequently, it has become increasingly important to 
demonstrate the concrete results of development assistance and to ensure the 
greatest possible effectiveness of current efforts. The Japanese authorities are now 
stressing the importance of having an effective evaluation system and are actively 
taking steps to further improve it. 

A report from the Administrative Reform Council published in 1997 empha- 
sised the importance of a mechanism to adequately take into account evaluation 
findings. The Basic Law for the Reform of Central Government Offices enacted in 
|une 1998 states that "all government offices must strengthen their objective eval- 
uation function and reflect the evaluation findings on policies”. 

In 1998 the Council on ODA Reform for the 21st century, an ad-hoc advisory group 
for the Minister for Foreign Affairs, released a report which included a call to further 
improve the evaluation system. Following this report, a working group was formed 
with a mandate to examine the evaluation function of the Japanese aid system. A 
series of recommendations for improvement was issued in March 2000. Among the 
main points raised was the need to establish a feedback system with a direct link to 
the policy making process, and a call for better co-ordination between the Ministry of 
Foreign Affairs, JICA and JBIC, and with other central government agencies. 

To address these major recommendations in greater depth, a new advisory 
group called the "ODA Evaluation Study Group” was formed (July 2000), composed 
of a wide range of actors from the development aid community luniversity profes- 
sors, aid experts from the private sector, international organisations, the business 
community, journalists, NGOs and officials from MoFA, ]1CA and JBIC). 

The Study Group report underlines the importance of ij strengthening policy- 
level evaluations and the improvement of programme-level evaluations, 
ii) strengthening its feedback system, in particular by the establishment of an 
effective feedback mechanism in Japanese aid organisations and iii) more 
effective ODA evaluation through promotion of cooperation and co-ordination 
among governmental organisations. The Ministry of Foreign Affairs has already 
taken steps in these areas. 

In order to ensure the swift dissemination of information to the public and to 
assure transparency, the Ministry of Foreign Affairs has started to promptly pub- 
lish outlines of evaluation findings on its web site, in addition to annual evaluation 
reports. 


* Adapted from Japan Ministry of Foreign Affairs: "Role and Responsibility of Central Evaluation 
Unit", submitted to the DAC Working Party on Aid Evaluation (WP-EV) Workshop, Tokyo 
26-28 September 2000. 
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workshops, media work, and suchlike. Since evaluation staff are not normally 
recruited with these skills in mind, for most agencies this will require bringing in 
qualified staff, or contracting work out to communication specialists. "We are profes- 
sional evaluators not professional communicators", as the Chair of the Working Party put it 
in his closing remarks. His advice was that agencies should bring in expertise 
where it is needed, not try and "do it all yourself". 



8. Conclusions 


The overall conclusion of the Workshop was that agencies need to take evalu- 
ation feedback much more seriously if they are to achieve the dual benefits of 
greater accountability and more effective learning. Many have already made con- 
siderable strides in gearing up their thinking and introducing more consistent and 
coherent feedback practices, as was evident from the presentations made. But 
most recognise that a lot more needs to be done. 

This is not an area where a blueprint approach would work. The diversity 
among agencies in their background, institutional context, and specific priorities, 
means that agencies need to work out their own solutions bearing in mind their 
particular circumstances, target audiences, and objectives. There were, however, 
many useful suggestions made at the Workshop that struck a chord, and which are 
clearly relevant to a good number of the agencies present, if not all. The most 
important of these are summarised in Box 6. These are presented in the form of a 
checklist of action points, with points divided into the following four categories, 
depending on the level at which they apply: 

• Central evaluation units 

• Agency-wide and senior management 

• Inter-agency 

• Partner countries 

This adds up to a substantial agenda for action, one that will present a consid- 
erable challenge for evaluation units, and for development agencies more 
broadly. However, the clear message that came through at the Workshop is that 
this challenge needs to be confronted, and in an imaginative and decisive way. 
Greater accountability and more effective learning are not options; they are 
essential if the development community is to deliver on the ambitious targets it 
has set itself, and if it is to retain the support of taxpayers at home, and the trust of 
partners overseas. The Workshop was seen as valuable step in facing up to this 
challenge. 
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Box 6. Checklist of action points to improve evaluation feedback 


Level 


Checklist of action points* 


CENTRAL 

EVALUATION UNITS 


Take steps to understand how learning happens within and outside 
the organisation, and identify where the blockages occur. 

Assess how the relevance and timeliness of evaluation feedback 
can be improved, and take steps to ensure this happens. 

Be explicit in identifying key audiences for evaluation feedback and 
the reasons for wanting to reach them, both in general, and in 
specific cases. 

Get to know target groups better to find out what they want from 
evaluations, how they use evaluation information, and how 
feedback systems can respond better to these demands. 

Develop a more strategic view of the how feedback approaches can 
be tailored to the needs of different audiences. 

Make sure the quality of evaluation outputs is up to standard - 
particularly from the point of view of brevity, clarity and presentation. 
Consider diversifying the range of approaches used to communicate 
with audiences, using "non conventional” methods where appropriate. 
Continue efforts to improve evaluation websites and intranets, 
recognising that ease of access and user-friendliness are key factors. 
Take steps to ensure that full disclosure of evaluation reports 
becomes the norm and that proper approval and notification 
processes are in place so that senior management or key partners 
are not caught unawares by controversial findings. 

Put much more effort into finding better ways of involving partner 
country stakeholders in evaluation work, including the feedback of 
evaluation lessons, recognising that language barriers are a key 
constraint. 

Recruit specialist staff where necessary to fill skills gaps, 
particularly in communications work. 


AGENCY- WIDE 
AND SENIOR 
MANAGEMENT 


Review whether the role of central evaluation units needs to be 
reconsidered in order to improve knowledge flows and the exchange 
of skills and expertise between evaluation, operational or policy 
departments. 

Consider whether the current positioning and status of evaluation 
units is limiting their effectiveness and undermining their ability to 
play a more active role in promoting learning and accountability, 
and take steps to rectify these problems if necessary. 

Ensure that senior management is fully behind the move to improve 
evaluation feedback, and is providing leadership in this area. 
Consider how evaluation lesson learning can be more effectively 
institutionalised, and identify where current feedback loops break 
down. 

Establish a routine of briefing senior management on evaluation 
lessons, and involving them in the communication of major findings. 
Take steps to increase the space and incentives for learning within 
the organisation (both from evaluations and other sources), and 
make it clear that failure to learn is no longer acceptable. 


_«J 
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Box 6. 

Checklist of action points to improve evaluation 
feedback ( coni .) 

Level 

Checklist of action points* 



Put more serious effort into making sure joint evaluation missions 
happen, and that problems caused by differing approaches and 
timetables are addressed. 



Reassess the future of the DAC Evaluation Inventory and develop a 
new strategy to ensure this becomes a more effective tool for lesson 

INTER- 


sharing. 

AGENCY 


Continue to use the DAC Working Party and other fora as a means of 
exchanging experience on evaluation feedback, and seeking out good 
practice. 



Support initiatives to improve training and capacity building in 
development evaluation work, and create links between evaluation 
professionals internationally. 



Put pressure on donors to coordinate their evaluation efforts more 
effectively and involve in-country stakeholders in a more consistent 
and meaningful way. 



Insist on being involved in evaluations right from the beginning of 
the process, and push for more ownership and control by local 
stakeholders. 

PARTNER 


Work to develop in-country capacity in monitoring and evaluation 

COUNTRIES 


and an open culture where evaluation is seen as a positive 
opportunity for learning. 



Decentralise responsibility for evaluation work, including feedback, 
beyond central ministries, bringing in civil society partners where 
appropriate. 



Adopt a policy of full disclosure of evaluation results as a means of 
demonstrating accountability and opening up possibilities for learning. 

* This list is not comprehensive hut aims to set down some of the main areas where action was called for at the Workshop. 





© OECD 2001 




Part I: Workshop Report 


Annex 1 

Workshop Programme 

DAC Tokyo Workshop on 

“Evaluation Feedback For Effective Learning And Accountability" 
26-28 September 2000 


Tuesday, 26 September 

09:00 Registration 
09:30 Welcome Addresses: 

Mr. Yutaka Iimura, Director-General of the Economic Cooperation Bureau, 

Ministry of Foreign Affairs of ]apan 
Mr. Niels Dabelstein, Chair of DAC Working Party 
09:45 Opening Session - Plenary 

Prof. Ryokichi Hirono (Chair): Workshop Aims and Objectives 

Theme: “Challenges- Sharing experiences to identify good practices, difficulties 
and problems” 

Mr. Geoff Barnard and Ms. Catherine Cameron: Key Findings from the 
Synthesis Report based on the Questionnaire and Interviews 

1 2:00 Lunch 

14:00 First Session - Plenary 

Theme: "Internal (learning): Feedback to improve effectiveness and efficiency of 
aid assistance". Topics: 

1 . Evaluation as a learning process (Germany: Mr. Axel Borrmann) 

2. Feedback for Better Decision-Making (Norway: Mr. Jan Dybfest) 

3. Role of Performance Monitoring and Evaluation in Managing for Results in USAID: 

Lessons Learned (USAID: Ms. Jean Du Rette) 

4. Role and responsibility of central evaluation unit (|apan: Mr. Mitsunori Shirakawa) 

1 4:40 Discussion 

15:00 Tea/Coffee 

I 5:30 Discussion in Working Groups 

Topics: 

1, Feedback for better policy 

2. Feedback for better practice 

"two working groups for each topic (4 working groups) 

I 7:30 Brief reporting back to Plenary 
1 8:00 Close 

19:00 Reception hosted by Mr. Yutaka Iimura, Director-General of the Economic Cooper 

ation Bureau, Ministry of Foreign Affairs in the "Ume" room (Hotel New Otani) 47 I 
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Wednesday, 27 September 
09:00 Second Session - Plenary 

Theme: “External (learning and accountability): Feedback of Evaluation results to 
parliament, media and the public". Topics: 

1 . Feedback to Parliament, Media and the Public (Netherlands: Mr. Rob D. van den Berg) 

2. Reporting Performance for External Accountability: DFID’s Public Service Agreement 
(UK: Dr. Colin Kirk) 

3. Dissemination Vehicles (Internet, report, video etc.) (Denmark: Mr. Niels Dabelstein) 

4. Involving Partners more effectively (Lao PDR: Ms. Khempheng Pholsena) 

5. Partner's Use of Evaluation Results (Sweden: Ms. Ann Marie Fallenius) 

09:50 Discussion 
10:00 Tea/Coffee 

10:30 Discussion in Working Groups. Topics: 

1 . Feedback to parliament, the media and the public 

2. Feedback to partner countries [two working groups for each topic - 4 working groups] 

12:00 Brief reporting back to Plenary 
1 2:30 Lunch 

1 4:00 Third Session - Plenary 

Theme: “Making Feedback More Effective". Topics: 

1. Dissemination: Ensuring Integration of Lessons (C1DA: Mr. Robert C. ]ones) 

2. Making Feedback more effective: the Case of France (France: Mr. Michael Ruleta) 

3. Learning from Evaluations: the World Bank's Experience (World Bank: 

Mr. Gregory K. Ingram and Mr. Osvaldo N. Feinstein) 

4. Results-oriented Management and Evaluation (UNDP: Mr. Sukehiro Hasegawa) 

1 4:40 Discussion 

15:00 Tea/Coffee 

15:30 Discussion in Working Groups. Topics: 

1 . New challenges in the changing aid landscape 

2. New tools for improved feedback ( e.g . Effective use of the Internet and other channels) 
* two working groups for each topic (4 working groups) 

17:30 Brief reporting back to Plenary 
I 8:00 Close 

Thursday, 28 September 
09:00 Closing Session - Plenary 

Theme: Overall Summary and Recommendations to improve feedback system for 
the better use of evaluation results: Mr. Geoff Barnard and Ms. Catherine Cameron 
09:30 Discussion 
10:30 Tea/Coffee 

1 1 :00 The way forward to Evaluation Feedback for Effective Learning and Accountability: 

Prof. Ryokichi Hirono, Mr. Niels Dabelstein 
Mr. Hans Lundgren, DAC Secretariat 

Concluding address: 

Mr. Yushu Takashima, Vice President, |apan International Cooperation Agency 
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Annex 2 

List of Participants to the DAC Tokyo Workshop 


1 

2 

3 

4 

5 

Chair 

Assistant to the 
Chair 

Chair of the DAC 
Working Party 
Vice-Chair of the 
DAC Working Party 

Mr. Ryokichi Hirono 
Mr. Kimihiro Kaku 

Mr. Ryujiro Sasao 
Mr. Niels Dabelstein 

Ms. Ann Marie Fallenius 

Professor Emeritus of Seikei University 
Professor of Toyo University 

Consultant 

Head, Evaluation Secretariat, Danida, Ministry of 
Foreign Affairs, Denmark 

Director, Department for Evaluation and Internal 
Audit, Sida, Sweden 

Member Countries 



6 

Australia 

Ms. Elizabeth Brouwer 

Director, Performance Information and Assessment, 
AusAID 

7 

Belgium 

Mr. Etienne de Beider 

Special Evaluator for International Cooperation, 
Ministry of Foreign Affairs 

8 

Canada 

Mr. Robert C. Jones 

Evaluation Manager, CIDA 

9 

Denmark 

Mrs. Ellen Buch-Hansen 

Advisor, Evaluation Secretariat, Danida, Ministry 
of Foreign Affairs 

10 

France 

Ms. Anne-Marie Cabrit 

Head of Development Activities Evaluation Unit, 
Ministry of Economy and Finance 

1 1 


Mr. Michael Ruleta 

Charge de Mission, DGC1D, Ministry of Foreign 
Affairs 

12 

Germany 

Mr. Axel Borrmann 

Senior Economist, Hamburg Institute of 
International Economics 

13 


Mr. Sigfrid Schroeder- 
Breitschuh 

Senior Policy Advisor, Evaluation Department, 
GTZ 

14 


Mr. Horst Breier 

Head of Evaluation, Federal Ministry for Economic 
Cooperation and Development 

15 


Mr. Ruediger Hartmann 

Division Chief, Evaluation Department, KfW 

16 

Netherlands 

Mr. Rob D. van den Berg 

Director, Policy and Operations Evaluation 
Department, Ministry of Foreign Affairs 

17 


Mr. Ted Kliest 

Policy and Operations Evaluation Department, 
Ministry of Foreign Affairs 

18 

New Zealand 

Ms. Jackie Frizelle 

Economist, Ministry of Foreign Affairs and Trade 

19 

Norway 

Mr. Jan Dybfest 

Assistant Director General, Head of Evaluation, 
Ministry of Foreign Affairs 
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20 

United Kingdom 

Mr. Colin Kirk 

Head of Evaluation Department, Department for 
International Development 

21 

United States of 

Ms. Jean Du Rette 

Chief, Program and Operations Assessment 


America 


Division, USAID 

22 


Mr. Constance A. Carrino 

Counsellor for Development Cooperation, 
Embassy of the United States of America 

23 


Ms. Rie Yamaki 

Embassy of the United States of America 

24 

European 

Mr. Simon R. J. Robbins 

Evaluation Manager, Evaluation Unit, Service 


Commission 


Commun Relex 

Non-Member Countries 

25 

Republic of Korea 

Mr. Taejoo Lee 

Director of Legal Affairs, Korea International 
Cooperation Agency! KOICA) 

26 

Bangladesh 

Mr. Kamrul Hasan 

Deputy Secretary, Economic Relations, Ministry 
of Finance 

27 

China 

Mr. Jin Xide 

Research Professor, Institute of Japanese Studies, 
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Annex 3 

Press Release 

by 

Prof. H irono, Chairman of the Workshop 

and 

Mr. N. Dabelstein, Chairman of the OECD/DAC 
Working Party on Aid Evaluation 

DAC Tokyo Workshop on 

“Evaluation Feedback for Effective Learning and Accountability” 

26-28 September 2000 

]apan hosted the DAC Tokyo Workshop on Evaluation Feedback for Effective Learning 
and Accountability on 26-28 September 2000. This was the first time Japan hosted a DAC 
Workshop on evaluation. This Workshop coincided with the establishment of the |apan 
Evaluation Society on 25 September. 

Eighty representatives from DAC countries, multilateral development agencies and 
banks, and participants from six Asian countries met to discuss and share experiences in the 
feedback of evaluation results. From |apan, the Ministry of Foreign Affairs, |apan Interna- 
tional Cooperation Agency, |apan Bank for International Cooperation, together with other 
government agencies, participated in this Workshop. 

Many development agencies are seeking to improve the efficiency and effectiveness of 
aid programs and aid policies. They recognise the central role of evaluation in these efforts 
and the importance of learning lessons and feeding back those lessons for the improvement 
of implementation and preparation for better aid programs. 

The Workshop participants shared experiences on communicating evaluation results to 
policy makers, operational managers, the public and the media in both donor and partner 
countries. 

Some highlights of their discussions include: 

It was generally recognised that present feedback systems should be further improved 
to meet the needs of wide range of stakeholders in donor and partner countries. Proper feed- 
back mechanisms should involve senior policy makers in the lesson learning process in 
development agencies. 

The importance of honesty and transparency in reporting evaluation successes and fail- 
ures was underlined. The public s understanding of the challenges involved in development 
cooperation will be enhanced by transparent evaluation feedback. I 
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Engaging partner countries in evaluation feedback systems is essential. The need for 
capacity building in partner countries is crucial. There was a wide consensus on the need for 
integrating evaluation feedback into the planning and monitoring programs and strategies in 
partner countries. This would improve development effectiveness and enhance accountability. 
The need for establishing and collecting the necessary indicators was also underlined. It was 
suggested that similar workshops should be held involving development agencies and 
stakeholders in partner countries as well as development agency evaluators. 

Without proper feedback, evaluation loses its life and value. The real value of evaluation 
should be assessed by the impact it has on the improvement of present programs and on 
future aid policies. 

Participants discussed the new challenges and opportunities in evaluation feedback 
brought about by the new aid landscape (such as conflicts, governance and human rights 
issues), new ways of delivering aid and new information technology. 

The OECD/DAC Working Party on Aid Evaluation will consider the results of the Work- 
shop and their implications at its next meeting on 22-23 November 2000. A publication with 
a summary of the Tokyo Workshop will be prepared. Individual agencies will take home 
experiences and lessons discussed at the Workshop for possible adaptation in their 
domestic context. 



P art II 

BACKGROUND SYNTHESIS REPORT 



Background Synthesis Report 

Prepared for the Tokyo Workshop 
of the DAC Working Party on Aid Evaluation 

26-28 September 2000 


Executive summary 

1 . This report concludes that there are now excellent opportunities for develop- 
ment agencies to improve their evaluation feedback for effective learning and 
accountability. 

2. It highlights the progress that has been made in the last decade and goes on 
to assess what is new in the context of the changing development assistance 
landscape in which agencies are now working. 

3. Evaluation departments are alert to the need for better evaluation feedback, 
particularly given their changing and widening area of responsibility. Many 
agencies have taken steps to address this, but most recognise the need to do 
more, and to be enabled to do more. 

4. Some of the key challenges facing evaluation departments include: 

• Resolving the contradictions between the dual roles of evaluation feedback 
- learning and accountability. 

• Responding to the shift from project evaluations to the new generation of 
broader-based evaluations focusing on themes or sectors. 

• Further improving dissemination strategies. 

• Finding better ways of institutionalising lesson learning 

• Reinforcing the reputation of evaluation units as a source of useful learning. 

• Increasing participation and stakeholder involvement in evaluations, without 
sacrificing core evaluation principles. 

• Responding to the growing problem of information overload. 

5. The main opportunities identified were: 

• Harnessing the Internet as a means of improving transparency and facilitat- 
ing lesson-learning within and between agencies. 57 I 
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• Tapping in to the new organisational management agendas of "knowledge 
management" and "managing for results" to enhance learning and account- 
ability. 

• Responding to the wider challenge presented to development agencies by 
the International Development Goals (IDGs). 

6. All three provide important opportunities for improving evaluation feedback 
so it becomes part of the "virtuous circle" of organisational learning, rather 
than being seen the "end of the line". 

7. Many agencies will have to step up a gear if they are to capitalise fully on 
these opportunities. However, if they are able to rise to this challenge, evalu- 
ation units have the opportunity to fulfil a much more central role within their 
organisations, and within the wider development process. 



1 . Introduction 


1.1. The story so far 

The challenge of how to ensure evaluation feedback for effective learning and 
accountability (hereafter "efela") is not a new one. Development agencies have 
been alert to this for many years, and the DAC and other groups have considered 
these issues in detail on a number of previous occasions. As a starting point for its 
work, the Study team was directed to three particular reports produced in the last 
1 5 years: 

• The 1985 DAC report 1 on "Good Feedback Practices", which identified four 
priority areas relating to the mechanisms for effective feedback, the audi- 
ences for feedback, the nature of evaluation findings, and the need for moni- 
toring action on feedback. 

• The 1990 DAC "Review of Donors’ Systems for Feedback on Aid Evalua- 
tion”, 2 which gives a progress report based on a review of donors' feedback 
systems. It separated these into two categories, dissemination mechanisms 
and institutional mechanisms, and went on to discuss what impact feedback 
was having in terms of changes in policies and procedures, and satisfying 
the information requirements of senior management. 

• The 1991 "Review of the Effectiveness of Feedback Mechanisms in Member 
States and the EEC Commission", which made fifteen recommendations, eight 
concerning feedback materials and seven concerning feedback mechanisms. 

The main conclusions and recommendations of the 1990 and 1991 reports are 
summarised in Annex 1 . They deserve rereading. Although progress has been made 
since they were written, most of the points highlighted are equally valid today. 

The 1990 report also provides a working definition of the term "evaluation 
feedback", which has been adopted for the purposes of this Report (see Box I ). 

1 .2. So what is new? 

Feedback practices have moved on considerably since these reports were 
written. Most agencies can point to a whole series of improvements and enhance- 
ments introduced over the past few years, especially in relation to the quality of 59 I 
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Box 1 . What is evaluation feedback? 

Previous reports have varied in their approach to defining evaluation feed- 
back. The 1990 DAC review (op. cit.) states that: 

"Evaluation feedback has been broadly defined as a dynamic process which involves the pre- 
sentation and dissemination of evaluation information in order to ensure its application into new or 
existing development activities... feedback, as distinct from dissemination' of evaluation findings, is 
the process of ensuring that lessons learned are incorporated into new operations." 

This report continues with the use of this definition, as it recognises that feed- 
back implies more than just dissemination of evaluation findings. It is about taking 
steps to ensure that evaluation lessons become part of a learning cycle within 
organisations. 


evaluation materials being produced, the recognition of the need for targeted 
information products, and the advent of the Internet as a new communication and 
information management tool. 

However, as the broader landscape of development assistance has evolved, 
the context of evaluation feedback has also changed. This has opened up new 
opportunities. But it has also put extra demands on evaluation units and created 
new expectations. 

In order to get a sense of how evaluation units regard these changes, and the 
challenges they now face, the efela questionnaire asked agencies for their views 
in two areas: 

• What they see as the most important emerging trends relating to evaluation 
feedback? 

• What they see as the areas of evaluation feedback most in need of improve- 
ment? 

In each case, agencies were asked to score various options according to their 
importance, on a scale of one to four (1 = not important, to 4 = very important). 
The results are summarised in Figure 1 , where the options are ranked according to 
the average scores given. 

Two important general points stand out; firstly, that evaluation units have a lot 
to deal with in terms of the number of emerging trends that they see as being 
important, and which are competing for their attention; and, secondly, that evalua- 
tion units are far from complacent about the need to do better on evaluation feed- 
I 60 back - they see many areas where improvements are necessary. 
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Figure 1 . Emerging trends in aid evaluation and feedback 1 


D1 : Areas most in need of improvement 

Wider distribution needed 
Better feedback during evaluation process 
More care to tailor outputs for specific target groups 
Type and format of outputs needs rethinking 
Better inter-agency lesson learning 
Feedback should be more timely 
Better sharing with stakeholders in recipient countries 
Steps to institutionalise lesson learning 
D2: Most important emerging trends 

Pressures from recipients for greater accountability 
Information overload 
Potential offered by new communications technologies 
Move towards participatory monitoring and evaluation 
Domestic pressure for greater accountability 
Changing way aid is delivered 

0 12 3 4 

Relative importance (average score) 

1 . The data presented here were collated from the responses to question D1 and D2 of the efela Questionnaire 
(see Annex 2). There were 29 respondents in total. The base for the averages was calculated from the total 
number of responses to each individual option within the two questions. 

Source: OECD. 



The emerging trends rated as being the most important were: 

• The changing way development aid is delivered. 

• Domestic pressure for greater accountability. 

• The move towards participatory monitoring and evaluation. 

In terms of the need for improvement, the areas rated as the highest priority 
were: 

• Institutionalising the process of lesson learning within the agency. 

• Sharing evaluation results with stakeholders in recipient countries. 

• Timeliness. 
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These were the top three responses in each case. Most of the other options 
were not far behind, however, reinforcing the view that evaluation feedback is an 
area of active debate and concern among most agencies. 

1 .3. Current practices: the starting point 

The efela questionnaire also yielded a great deal of useful information on 
current agency practices and thinking. The picture that emerges is complex. As 
well as differences in policies and approaches, it also reflects the diverse back- 
grounds and institutional contexts of the agencies involved in the survey. Agencies 
are not all starting in the same place in their approach to evaluation feedback. This 
makes a big difference in terms of the mandate and objectives of evaluation units, 
their lines of reporting and accountability, and their relative priorities. 

A broad typology of DAC Working Party Members is presented inTable 1, 
along with some of the implications and issues arising in relation to evaluation 
feedback. 

Because of this diversity, it is difficult to draw generalisations that apply 
equally across all agencies. Instead, the emphasis in this Report has been on high- 
lighting emerging issues, and contrasting different approaches and experience. 

However, a number of broad headline messages did emerge from the Question- 
naire, and these are summarised in Box 2. For a more detailed analysis of the 
results, including a synopsis of the comments provided individual agencies, 
see Annex 2. 

1 .4. Structure of the report 

Evaluation feedback is a multi-faceted subject that impinges on many 
aspects of evaluation thinking and methodology, and cuts across into many other 
related areas. This was demonstrated in the questionnaire responses and rein- 
forced in the interviews with individual agencies. Both showed the wide range of 
perspectives on the subject, and the rapid evolution of current thinking. 

This makes it difficult to draw a boundary around the subject, or divide it into 
neat categories. For the purposes of this Report, which is intended as a stimulus 
for discussion at the Tokyo Workshop, the topic is considered under two broad 
headings: 

• Challenges - the current issues and dilemmas that agencies face when 
seeking to improve their evaluation feedback practices. 

• Opportunities - areas where new thinking is emerging and new possibilities 
are opening up for improving evaluation feedback practices and linking 

I 62 efela with other objectives and initiatives in the development policy arena. 
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Table 1. Typologie of DAC Working Party Members and Observers 


Type* 

Agencies 

Implications and issues arising 

Co-ordinating agency 

Development Assistance Committee 
of the OECD (DAC) 

Authoritative policy guidance 
Periodic critical reviews 
Centralised statistics and reporting 

Bilateral donors 

Australia Netherlands 

Austria New Zealand 

Belgium Norway 

Canada UK 

Denmark US 

Finland Portugal 

Ireland Spain 

Italy Sweden 

Luxembourg Switzerland 

Primary accountability to domestic 
parliament and taxpayers 
Usually grant financing 
Fewer commercial considerations 
Fairly stable unitary structure 
Disclosure not usually a major issue 

Bilateral donors with 

Japan - MoFA, JICA, JBIC 

Foreign ministry, technical assistance, 

a tripartite split 

France - MFA, MEFI, AFD 

financial aid split three ways 

of Ministries 

and implementing 

agencies 

Germany - BMZ, KfW, GTZ 

Primary accountability to domestic 
parliament and tax payers 
Tripartite structure can lead to longer 
reporting lines, need for increased 
co-ordination with varying objectives 
Mix of grant and loan funding 
Disclosure practices vary 

Multilateral agency 

European Commission 

Accountable to Member States, European 
Parliament and Court of Auditors 
Accession states issue and reform of 
Commission, so size and structure likely 
to change 

Grant funding (loans are channelled 
through European Investment Bank) 
Could be useful linkages with bilateral 
DAC members who are also EU members 

Observers: 

African Development Bank 

Loan based funding (hard/soft) 

Multilateral 

Asian Development Bank 

Accountable to broad body of 

Development Banks 

EBRD 

IADB 

IMF 

World Bank 

shareholders 

Disclosure issues, commercial 
considerations, can affect evaluation 
feedback policies 

Other agencies 

UNDP, Czech Republic, Korea, Mexico 


* DAC Members' organisational structures are more fully broken down in DCD(99)6. Tfie classification here has been simplified from an 
efela perspective. 


In each case the report aims to draw out some of the main areas of thinking 
and learning and points to where good practice - or at least "better practice” - is 
beginning to emerge . 3 
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Box 2. Headline messages from the questionnaire* 

Strategy and responsibilities 

• Fourteen of the twenty-nine respondents have a formal strategy for dissem- 
ination and feedback of evaluation results, and most of the other agencies 
have elements of a strategy in place, or are currently working on one. 

• Internal agency audiences (policy makers and senior managers) are seen as 
a somewhat higher priority than stakeholders in partner countries; the 
media and the general public are seen as the least important, 

• Most evaluations units have prime responsibility for feedback for learning pur- 
poses within the organisation, whereas in the case of feedback for externa] 
accountability this responsibility is more often shared. Disclosure is an issue for 
some agencies. 

• Relatively few agencies employ staff or consultants specifically for dissemi- 
nation and feedback work - more often this is incorporated into general 
staff responsibilities. 

• Only a few agencies have a specific budget for dissemination and feedback 
work. 

Feedback and dissemination mechanisms 

• Evaluation reports are seen by half the agencies as the main vehicle for dis- 
seminating evaluation results. 

• All agencies now require that evaluation reports contain short summaries, 
although the length and format of these varies considerably. 

• Separate summaries are now much more common, but only a third of agencies 
produce these on a systematic basis. 

• Feedback of interim results during the evaluation process is seen as a priority 
by half the agencies. 

• Two thirds of agencies take active steps to ensure evaluation reports are fed 
back to in-country partners and stakeholders. 

• Agency intranets and web sites are now being used by about a third of agencies 
for disseminating evaluation results; another third have plans to do so. 

• A wide variety of other channels are also used for dissemination and feed- 
back of evaluation results. The most frequently mentioned were: annual 
reports; staff workshops, seminars and training events; and summary 
reports covering particular sectors/regions. 

• Various mechanisms are in place to make sure that lessons from evaluations are 
taken properly into account. The most common of these is the requirement for 
senior management to respond to evaluation findings and recommendations. 

• Arrangements for inter-agency lesson learning are quite patchy. Joint evalu- 
ation missions are rare, and tend to be contracted out to consultants. 
Although 15 agencies contribute to the DAC Evaluation Reports Inventory 
on a frequent basis, only 5 report that they use it frequently. 


* See Annex 2 for a more detailed summary of the Questionnaire results. 
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Notes 


1 . "Good Feedback Practices” - DCD/DAC/EV(85). 

2. "A Review of donors' systems for feedback on aid evaluation" - OECD/GD(90|. 

3. Relatively few agencies are ready to nominate their procedures as being "good prac- 
tice”, since, despite the progress that has been made, most are very much aware of the 
shortcomings of their existing mechanisms and practices. 
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Getting evaluation feedback right is not easy. Evaluation units face a whole 
series of challenges. And as they make progress in some areas, new demands and 
complexities open up in others. For small agencies and those with limited 
resources, this makes for a daunting agenda. Given the range of challenges to 
confront, and the pressures to perform, even the largest agencies are being fully 
stretched. 

2.1. Dual objectives: learning and accountability 

The title of the Tokyo Workshop encapsulates the dual objectives of evalua- 
tion feedback: learning and accountability. There are many crossovers between 
the two, both actual and potential. But they are not the same thing. 

This point is now well recognised and has its parallels in other areas. The 
recent DAC Report on Results Based Management (RBM) 1 notes that performance 
information serves two primary uses: improving internal management (managing 
for results), and external reporting (accountability for results). The report goes on 
to observe that as experience grows, the potential for conflict between its two 
intended uses is emerging, as they may not be entirely compatible. Information is 
required at different levels, with different data collection and analysis 
approaches. So there is a growing dilemma for donor agencies. 

Box 3 illustrates how the target audiences for learning and accountability pur- 
poses differ. In both cases the range of audiences has expanded in recent years, 
especially with the new emphasis on accountability, partnership with recipient 
country stakeholders, and better coordination between agencies. But although 
there are some overlaps, for the most part the audiences are quite distinct. This 
has implications for the kinds of products and mechanisms needed to reach differ- 
ent target groups, and underlines the message that one approach cannot be 
expected to suit all audiences. 

It also has implications for how the whole evaluation process is managed: 

• Where accountability is the priority, the traditional virtues of rigour, inde- 
pendence, replicability, and efficiency tend to be the primary concerns. 
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Box 3. Target audiences for evaluation feedback 

In the last decade, as the scope and ambitions of evaluation have expanded, 
the range of target audiences for evaluation feedback has increased. The charac- 
teristics and demands of these audiences vary, as does their relative importance 
in accountability and learning terms. 


Primary audiences 
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purposes 
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purposes 
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• Where learning is the priority, the emphasis is more likely to be on achiev- 
ing "buy-in" from stakeholders, focusing on the process, and creating space 
to make sure that experience is properly discussed, and lessons drawn out. 
Timeliness is also a more important factor, since the desire is to pass on les- 
sons now, not in a year's time when the final report is eventually published. 

These two objectives are not necessarily incompatible, and many agencies 
are looking hard at how to find compromise methodologies that satisfy both sets 
of criteria. But they are sufficiently different to merit separate consideration. 
Feedback strategies that blur the distinction between the two run the risk of fall- 
ing short in both areas. 

2.2. The move away from project-level evaluations 

Over the past decade, there has been a significant shift in the way develop- 
ment assistance is delivered and measured . 2 For many agencies, the traditional 
project-based approach is giving way to broader country programmes with the- 
matic initiatives and sector-wide approaches . 3 This has had major implications for 
the way evaluations are conducted and for the methodologies and management 
approaches used . 4 It also has changed the whole context for evaluation feedback, 
increasing its importance within the agency . 5 

From an evaluation perspective, individual projects have the advantage of 
being discrete and relatively easy to define. They generally have a reasonably 
clear boundary in terms of their geographic and sector focus, their timeframe, and 
the stakeholders involved. A well-managed evaluation has a good chance of iden- 
tifying the most obvious direct outcomes of the project, and the lessons that have 
emerged from it. The process is relatively neat and contained. Results can be fed 
upwards to satisfy accountability requirements. If the necessary consultation 
mechanisms are in place, the stakeholders most closely involved in the project 
can contribute to, and benefit directly from, the lessons emerging. 

The problem with project evaluations is that they can be too discrete. Les- 
sons tend to be very specific, making it hard to generalise from them. And 
because evaluations are normally carried out at the end of the project cycle, the 
lessons learned may be too late to affect that particular case. 

With broader-based evaluations the situation tends to be reversed. The 
boundaries are much harder to define, there are far more stakeholders involved, 
the focus is more diffuse, and it is often harder to identify outcomes in a way that 
they can clearly be attributed to a particular donor's intervention. This makes the 
process much more complex, particularly when it comes to demonstrating aid 
effectiveness. It means that evaluations can take longer, and tend to be more staff 
I 68 and resource intensive. 
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The attraction of broader-based evaluations is that the lessons learned are 
likely to be more widely applicable, both in policy and operational terms. It there- 
fore makes sense to invest more in repackaging them for different audiences, and 
in ensuring they are widely communicated. By involving a wider range of stake- 
holders there is also a much better chance of building lesson-learning into the 
evaluation process itself, and of sharing experience between donor agencies, gov- 
ernments and civil society partners. 

This shift in focus is taking evaluation units into new territory where much is 
still to be learned. But it is also increasing the potential influence of evaluation 
work, and offering the prospect of a much more effective learning process. 

2.3. The dissemination challenge 

Making information available in an appropriate format for its intended audience 
is one of the most basic rules of good evaluation feedback. Progress has been made 
on this front in the last few years, and the standard of evaluation products has 
improved substantially across the board. In Annex 3, some of the main types of evalu- 
ation products are described, using examples from a cross-section of agencies. 

Some agencies are further ahead than others in terms of the effort they put in 
to producing tailored evaluation products for different audiences. DANIDA, for 
example, on the basis of an evaluation of its health sector programme in Bhutan, 
produced a full report in English (150 pages); a 30-page booklet in Danish, a 4-page 
evaluation summary, and a video aimed at the general public. 6 lapan has also pro- 
duced a video outlining the Japanese aid programme and explaining the evalua- 
tion process using examples from India and Fiji. Other agencies have a more 
standardised approach, and rely on the main evaluation report, usually with an 
executive summary, as the chief communication vehicle. 

Producing well-edited and professional-looking evaluation products is an 
important step. But it is not enough, by itself, to ensure they are read and used. 
Agencies are becoming increasingly aware of other factors that come in to play: 

• Timeliness: a beautifully produced synopsis of a project which finished sev- 
eral years ago will be of little interest to most audiences. Any lessons that 
emerged have probably already been learned, or will be seen to be past 
their "sell by date". 

• Relevance: products need to respond to current agendas, debates and 
needs, otherwise they are likely to be dismissed as irrelevant. 

• Marketing and distribution: evaluation products are of little use if they sit in 
boxes (or databases), or get sent to out-of-date mailing lists. Though most 
agencies recognise this as an issue, relatively few are well geared up in this 

area. Some make special efforts and organise press conferences to "launch" 69 I 
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evaluation reports. 7 More commonly, by the time the final product emerges, 
the momentum behind the exercise has fallen away, and the staff involved 
have moved on to other tasks. Though a great deal of effort and thought 
may have been invested in producing the product, marketing and distribu- 
tion is often left to chance or is treated as a routine exercise, with very little 
systematic attempt to target products intelligently. 

• Credibility: users judge information based on the credibility of the messen- 
ger, not just the quality of the message. Although they may have a good rep- 
utation in terms of their professional competence as evaluators, evaluation 
units generally have some work to do in establishing their status as good com- 
municators, or as potential sources of cutting-edge new thinking. 

All four of these factors pose challenges for evaluation units, challenges that 
can only be addressed through concerted action. The first two are perhaps the 
most critical, however. Evaluation products that are seen as both timely and relevant 
will generate their own demand, and are one of the best ways of helping build the 
reputation of the evaluation unit as a source of information that is worth tapping. 

2.4. Institutionalising lesson learning 

influencing policy and practice is one of the key objectives of aid evaluation. 
The way this happens in practice, however, is far from being an exact science. 
Agencies are very different in their culture and the way they work, and have devel- 
oped a range of internal mechanisms to try to institutionalise this. Some recent 
examples include: 

• Following the recent reform of Japan’s evaluation system it is proposed that 
explicit policy level evaluation will be introduced. This will include reviewing 
medium term policy, country assistance programmes and particular themes. 8 

• DFID has introduced the concept of Performance Scoring and further devel- 
oped its' Performance Reporting Information System for Management. In addi- 
tion external reporting requirements for the Public Service Agreement have 
focused attention on the need for improved country level information. 

• BMZ has a systemised follow-up approach to their evaluation work, where 
after a year evaluation recommendations are tracked to see whether or not 
they have been implemented. France did this as a one-off exercise in 1998 
and may systemise this process. JICA and KfW have a five-year follow-up pro- 
gramme; |BIC conducts follow-up after two and seven years. 

• In SIDA, the central evaluation and audit functions have been combined in 
an independent department, under a joint director who is responsible to 
the Board. The crossover between audit and evaluation functions has 
reportedly been helpful, as it has encouraged a more focused and targeted 
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response to evaluation recommendations. Senior management, for exam- 
ple, is required to draft a specific response to evaluation reports.** 

• EC - the Quality Support Group is a relatively recent development and has 
attracted interest and participation at a senior level, significantly raising the 
profile of evaluation feedback work at a senior management level. With the 
recent expansion in the responsibilities of the Evaluation Group as a part of 
the DG re-organisation this has been useful. Ways and means of improving 
evaluation quality and feedback are under review, including an idea for a 
harmonised rating system. 

• The World Bank continues to develop the concept of results based manage- 
ment, including the development of a Corporate Scorecard, and the recent 
production of Fast Track Briefs to inform senior management. It is in the 
process of revising its use of Management Action Records, a feedback loop 
whereby key action points for management emerging from an evaluation 
are identified. These are tracked subsequently to make sure they have 
been implemented. 

Despite these and other efforts to systematise the procedures involved in 
making sure evaluation lessons are learned, there remains a substantial element 
of randomness in the process. Some evaluations prove to be much more influen- 
tial than others and this is often due to factors that are beyond the control of eval- 
uation units. In this respect, evaluation is similar to development research, which 
also seeks to influence policy and practice, but is equally susceptible to factors 
outside the immediate control of the researchers themselves. 

Some of the reasons for the divergence between theory and reality, and the 
implications for evaluation units, are explored in Box 4. While there are no easy 
solutions to this problem, recognising the complexity of the policy process is an 
important first step. 

2.5. Process and politics 

In planning feedback strategies, there are a whole series of dilemmas and 
internal tensions that evaluation units have to come to terms with, which are to do 
with the process and politics of dealing with evaluation information. At times, 
these can be quite sensitive, though their relative importance will vary depending 
on the specific circumstances. Some of the more important of these tensions are 
summarised in schematic form below. 

None of these dilemmas are easy to resolve, and in particular circumstances 
agencies have come to different conclusions on what is the best way forward. 
These are all active areas of debate, however, where sharing of experience would 
be welcomed. 
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X 

/ Being open and willing to 

N. deal frankly with problems 


Disclosure 


“Cleaning” reports to reduce 
the risk of information being 
taken out of context 


— 

Releasing interim results 
quickly to maximise 
\ opportunities for learning 


Timing 


Releasing results in slower 
time with full consensus from 
all stakeholders if possible 


Broadening out consultation 
processes to include a wide 
range of stakeholders 


Consultation 


Keeping the process 
manageable and getting the job 
done in a reasonable timeframe 


Decentralising evaluation 
work to operational units that 
are closest to the action 


Devolution 


Retaining a strong central 
function to prevent lesson- 
learning being fragmented 


< Delegating responsibility 
to other units to encourage 
buy in and lesson learning 


Quality control 


Maintaining explicit oversight 
to ensure standards of quality 
and rigour are maintained 


| 72 


Box 4. The feedback fallacy 


One of the primary aims of evaluation feedback is to ensure that evaluation 
work influences policy and practice. Given this objective, it is helpful to reflect on 
the process by which policies are set, and are converted into practice. Behind 
much evaluation work there is an implicit assumption that this process happens in 
a rational and broadly linear fashion, which, in a highly simplified way, can be rep- 
resented as a series of steps: 


Defining 
the nature 
of the issue 
to be dealt with 


Identifying 
possible courses 
of action 


Weighing up 
the pros and cons ' 
of each alternative / 


Choosing 
the option 
which offers 
the best solution 


Implementing 
the policy 
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Box 4. The feedback fallacy ( cont .) 

The implication is that by feeding in credible, well-expressed information, 
at the right point in this chain, evaluation findings will have a direct influence 
on policy and practice. The reality, of course, tends to be more complex. Eval- 
uation lessons are just one of the ingredients in what has been termed the 
"policy soup". Despite management efforts to systematise lesson learning and 
feedback loops, there are a whole range of other factors which come into play 
that can have a big impact on how policies are determined, and whether or not 
they get implemented. Some of the most notable of these are: 

• Political factors - at every level, from internal agency power struggles to 
global politics. 

• Vested interests - which can promote or block change, depending on who 
stands to gain. 

• Chance - a report arriving at the right place at the right time, for example, 
can have a disproportionate impact. 

• Bureaucratic inertia and aversion to risk. 

• The persistence of conventional wisdom - even when proved wrong. 

• Inspired leadership - which at times can shift thinking dramatically. 

• Bandwagons - which can have a major influence (good and bad), once they 
get rolling. 

• Pressure to spend - which can result in evaluation lessons being heard 
selectively, given the pressures on staff to turn projects around quickly, and 
get funds disbursed. 

Evaluation Departments have little control over most of these factors. This 
helps explain why high quality evaluation results, even when they are well com- 
municated, may not be enough to trigger significant change. This is a fact of life 
within agencies; it is even more the case when the influence of evaluation results 
on recipient countries and the broader aid community is considered. This needs 
to be borne in mind both when planning feedback strategies, and in assessing 
their effectiveness. 

If the objective is to maximise impact on policy and practice, a more sophisti- 
cated analysis may be needed of how change is likely to occur, and what factors 
may be blocking it. Looking back at their experience, most agencies can pick out 
cases where evaluation results were particularly influential for one reason or 
another. The challenge is to look forward, and identify where timely and targeted 
interventions can have a greater influence in the future. 


* This box draws on R. Sutton (1999), “The Policy Process: an Overview ", ODI Working Paper No. 118. 
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2.6. The role and status of central evaluation units 

Underlying much of this analysis there are important issues relating to the 
role and status of central evaluation units. This varies enormously between agen- 
cies, and is shifting over time, not always in a helpful direction. Evaluation units 
are under pressure on a number of fronts: 

Changing Role: Over the last decade, central evaluation units have shifted to do 
much more policy-focused, strategic level work. Project level evaluation and 
feedback are now increasingly the domain of departments doing the direct 
spending. The role of the evaluation unit in such cases can become unclear. 
Do they provide professional oversight? Do they respond to ad hoc queries? It 
is not always obvious where responsibility lies. There is a danger that evalua- 
tion feedback can become fragmented, with a tendency for gaps and duplica- 
tion to emerge. There is also a risk that, in the absence of central evaluation unit 
bench-marking, a quality versus quantity trade-off may emerge. 

Evolving trends: As the way of delivering development assistance has changed, so 
have methodologies to evaluate it. "Blueprint" infrastructure projects, integrated 
rural development, structural adjustment, programme aid, sector wide 
approaches, medium term expenditure frameworks, humanitarian assistance, 
conflict resolution, the good governance agenda: in the last thirty years a whole 
series of different approaches have been tried. The aid delivery process has 
become ever more complex and evaluation units have been expected to deliver 
with each new approach. 

Increased workload, same resources: The recent DAC report on Results Based 
Management notes that as evaluation units take on these broader functions there 
may be competing demands on staff time and resources. Evaluation units now 
talk the language of performance management and senior management has often 
delegated the role of supplying information for this purpose to them. But the 
extent of their existing responsibilities may not have diminished, so some evalu- 
ation units are finding themselves being stretched in a new direction without the 
resources to deliver effectively. There is also a risk of de-emphasising traditional 
evaluation work. 

Status: Some central evaluation units face additional challenges due their status 
within their organisation; the "Cinderella syndrome". This may be demon- 
strated by the usual resource constraints of both funding and staffing. It can 
combine with a reluctance to hear what evaluation departments have to say. 
The department may be perceived as the policeman or auditor within, an 
impediment to delivering the programme. Or it may be regarded as an aca- 
demic ivory tower, remote from the real action. The problem can be self- 
perpetuating since the highest calibre agency staff may be reluctant to move 
to evaluation units, which are seen to be remote from the cutting edge of 
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policy or delivery. Even the physical location of evaluation units can be an 
issue: those that are located at the "centre'' of the organisation are more likely 
to be taken seriously than those situated in remote offices. 

These, too, are realities that cannot be changed overnight. They are areas 
where leadership from the top will usually be required in re-scoping the work of 
evaluation units and raising its status and resourcing over a period of time, so it 
can perform more effectively as a focal point for learning and accountability. 

2.7. Encouraging participation and ownership 

At one time, the main function of monitoring and evaluation systems was to 
provide information for donor requirements, with the needs of recipient country 
partners being regarded very much as a second order consideration. This balance 
is shifting. In the efela Questionnaire, twenty agencies rated partner country stake- 
holders as being "important" or "very important" target audiences. Although this 
group still came behind internal audiences in the priority ranking, some agencies did 
comment that their scoring reflected the reality, rather than the ideal situation. 

Most agencies reported that they take active steps to ensure that evaluation 
reports are fed back to in-country partners and stakeholders , 10 but the majority 
also accept that they are not doing enough. Part of the problem may be the per- 
ceived lack of demand within recipient countries. It is significant that pressure 
from recipient countries for greater accountability was ranked lowest amongst the 
emerging trends in the Questionnaire. So, for many agencies, increased involve- 
ment of stakeholders is being promoted because it seen as axiomatically a good 
thing, rather than as a response to specific demand. 

At the same time, the new wave of participatory thinking is driving the pro- 
cess forward. With the increasing emphasis on the need to incorporate the per- 
spectives of stakeholders, particularly the poor, participatory monitoring and 
evaluation (PM&E) has emerged as the logical next step. It involves local people, 
development agencies, and policy makers deciding together how progress should 
be measured, and results acted upon Advocates of this approach argue that 
PM&E provides an opportunity for development organisations to focus better on 
their ultimate goal of improving poor people s lives. By broadening involvement 
in identifying and analysing change, a clearer picture can be gained of what is 
really happening on the ground. For those involved, it can also be a very empow- 
ering process, since it puts them in charge, helps develop skills, and shows that 
their views count . 11 

Though a lot of agencies are talking about PM&E, and have adopted it at the 
rhetorical level, relatively few have had any hands-on experience of applying it in 
practice . 12 Encouraging participation in the evaluation process is a natural next 
step for projects which have been planned and implemented in a participatory 75 I 
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Box 5. PM&E: new communication challenges* 

Participatory Monitoring and Evaluation (PM&E) is posing a whole set of new 
challenges for development agencies. Some of these have an important communi- 
cation angle. 

• Bridging communication gaps - participatory approaches are all about 
communication between different stakeholders. This process can be diffi- 
cult when the status and background of the stakeholders varies widely, and 
when they are speaking in different languages. Even if they share the same lan- 
guage and dialect, use of technical expressions or evaluation jargon can be a 
real turn-off for those unfamiliar with the terminology, and instead of encourag- 
ing buy-in, may be actively disempowering. These communication gaps can be 
bridged, given time and the right attitudes and behaviour. But if they are 
ignored, meetings can end up achieving little it terms of real dialogue. 

• Aggregating lessons - successful PM&E can generate a great deal of locally 
relevant lessons. But how can this be aggregated to produce broader con- 
clusions that stand up to analysis and can feed into policy, without loosing 
the detail and diversity which is the essence of understanding individual 
situations? This dilemma was faced in the recent "Consultation with the 
Poor” exercise conducted by the World Bank, in which information from par- 
ticipatory work at 220 sites around the world was pooled in order to draw 
out some broad conclusions. New techniques are being developed to help 
with this, including databases designed to discern common patterns from 
large amounts of qualitative data, or from pictures and diagrams. Others 
argue, however, that trying to aggregate local data is misleading, and that 
the answer is to make policy more responsive to diverse local situations. 

• Reaching out - where PM&E is generating new learning, questions arise as 
to who needs to know and how those messages need to be communicated. 
If they are to have any impact beyond the immediate stakeholder group, 
they will need to be communicated in a concerted way. This may require 
adopting an advocacy role. Or lessons may need to be given scientific 
respectability to be taken seriously. However it is done, an investment in 
skills and resources may be needed. If the process is generating controver- 
sial conclusions that challenge existing power structures, funding agencies 
may need to think carefully about how far their role can extend, and how 
closely they should be associated with the political forces that genuine par- 
ticipatory processes can unlock. 

• Communicating in reverse - feeding information upwards from the local level 
poses one set of challenges. But there may be an equal need to communicate 
in the other direction. It is very common for stakeholders involved in evaluation 
exercises to have very little idea about why they are being consulted or even 
what the purpose of the project was in the first place. Without this knowledge, it 
is hard for them to engage in the process fully or offer a considered view. And 
once the consultation phase is over, stakeholders are rarely informed of the 
outcome of higher level decisions that are subsequently taken. Both are seri- 
ous flaws that prevent real lesson learning and accountability. 


* Based on personal communication with \ohn Gaventa, head of the IDS Participation Group. 
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fashion; it is much more of a shift in thinking for projects that have not been carried 
out in this way, and where the basis of trust and partnership is weaker. 

A recent review of attitudes to PM&E among DFID advisers concluded that 
"the concept of PM&E is broadly accepted but narrowly understood". It found that par- 
ticipation was usually limited to consultation with beneficiaries at particular stages 
in the monitoring and evaluation cycle. There was relatively little appreciation of 
the need to engage other actors, such as governments, the private sector, and 
trade unions, and there were few examples of PM&E leading on to genuine stake- 
holder participation in decision-making and change. 13 

Much remains to be learnt about how PM&E approaches can be applied sys- 
tematically, and how they can be integrated in an intelligent way with other moni- 
toring and evaluation efforts. Some of the challenges that PM&E is creating from a 
communication and feedback perspective are reviewed in Box 5. 

Two of the obstacles that prevent greater involvement by stakeholders in 
monitoring and evaluation, and helps explain the weak demand for it, are the lack 
of capacity in many recipient countries, and the absence of a strong accountability 
and learning culture. A number of agencies are beginning to focus on these. A joint 
African Development Bank and World Bank workshop on in-country capacity building 
was held in Abidjan in 1 998, for example. The World Bank is also looking at options for 
developing regional "centres of excellence" in monitoring and evaluation. 

As part of its partnership agenda, DFID has taken a first step in translating 
some new country strategy papers into the language of recipient countries, seek- 
ing to promote understanding and hence ownership. S1DA had worked to close 
the feedback gap in-country, for example, in Botswana and Namibia. Such mea- 
sures are part of a long-term process. It is clear that a great deal of work needs to 
be done to turn the new rhetoric of stakeholder involvement into a reality. Even 
dealing with the very basic issue of translating evaluation materials into local lan- 
guages will be a major task. But until this becomes a routine practice, it is hard to 
see how widespread stakeholder involvement can take hold. 

2.8. Responding to information overload 

This final challenge is perhaps the most mundane, but is nonetheless impor- 
tant. For development agency staff, and many of those they work with, information 
overload is a serious and growing problem. A decade ago it could be measured by 
the size of the stack of unread documents piling up in your office. Nowadays, 
although the piles of documents persist, it is the number of unread messages in 
your e-mail inbox that provides the most graphic (and alarming) reminder. Though 
few would doubt that the new generation of information and communication tech- 
nologies (ICTs) are part of the solution to better information exchange, they are 77 I 
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Box 6. ICTs - a double-edged sword 

The introduction of new information and communication technologies (ICTs) 
has created problems, as well as offering solutions. For example: 


E-mail is allowing cheap and almost 
instantaneous information sharing 
between individuals and groups 



E-mail communication is often less 
systematic and well thought out than 
traditional paper-based approaches 


E-mail makes it easy to copy 
people in when sending messages, 
just in case it is of interest to them 



People are deluged with “just in case” 
information, and cannot distinguish 
what is really important 


E-mail lists and discussion fora 
are allowing groups with shared 
interests to exchange ideas easily 



It is easy to get buried - deleting 
messages is quicker than sorting out 
what is relevant or un-subscribing 


Web sites are providing an 
ever-growing wealth of information 
and data on almost any topic 



How do you know where to start, 
an how do you assess the quality 
of the information that is out there? 


also becoming part of the problem. Some of the manifestations of this contradic- 
tion are explored in Box 6. 

Software packages and work practices are evolving to help deal with the prob- 
lem. But most would agree that at present the overall problem of information 
overload is getting worse, not better. This is a reality that evaluation units need to 
deal with in planning their communication strategies. 

One interviewee at the InterAmerican Development Bank referred to a more 
specific variant on this, "lesson overload". He was referring to the proliferation of 
development lessons, coming at operational staff from all directions: from policy 
departments, research organisations, pressure groups, and other sources. Amid this 
cacophony of messages, it is not always easy to make evaluation lessons heard. 

Experience from a range of information management contexts 14 suggests that 
some of the key lessons in combating information overload are that: 

• Editing matters - users appreciate information that is short, well drafted, 
and to the point, nd are increasingly impatient with lengthy documents that 
have to be struggled through in order to extract key messages. 

• Less is more - bombarding people with more and more information can be 
counter-productive. It is better to be selective. Recipients are more likely 


©OECD 2001 


Copyrighted i 




Part II: Background Synthesis Report 


to pay attention to messages if the sender is seen to be responsible and 
thoughtful in what they send out. 

• Highlights - short e-mails and other bulletins flagging new information that 
is available, and showing people where they can find it, can be more effec- 
tive than sending lengthy full text documents. 

• Demand pull - people are generally more attentive to information they have 
requested themselves, and which matches their specific needs. But demand 
needs to stimulated through good publicity and clear signposting. 

• Listening to users - it is easy to fall into the trap of assuming you know what 
users need, or treating them as a uniform group. People's information needs 
and habits vary, and they change over time. Regular audience research is 
needed to stay in touch with users, and allow communication strategies to 
be fine-tuned to match their needs. 
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Notes 


1. Results Based Management in the Development Co-operation Agencies: A Review of 
Experience, DCD/DAC/EV(2000)3. 

2. The changing way aid is delivered was ranked as the most important emerging trend in 
the efela Questionnaire; it was rated as "important 1 ' or "very important" by 2 I of the 
24 agencies responding. 

3. USAID, for example, no longer has projects, but plans its work around "strategic objectives". 

4. See, example, OECD (1999) "Evaluating Country Programmes”. Report of the 1999 
Vienna Workshop. 

5. Agencies are at different stages in their response to this shift. Some, such as USAID, are 
now putting all their emphasis on broader-based country-wide or thematic evaluations. 
But most are maintaining a mix of approaches, and see a continuing need for project 
evaluations. 

6. The DAN1DA video was produced by an independent film crew and tracks the progress 
of the multidisciplinary team that undertook the evaluation. It takes a frank and broad- 
based approach, looking at the cultural background to healthcare in Bhutan, and the 
relationship between the modern approaches being pursued through the project, and 
the traditional forms of medicine that exist in parallel. 

7. This is the case in the Netherlands and Denmark, for example. 

8. Summary of the Report on Reform of lapan’s ODA Evaluation System, March 2000. 

9. This requirement is now quite widespread, according to the efela Questionnaire. 
20 agencies indicated that senior management is expected to respond to evaluation 
findings in one way or another. 

10. J BIC now holds two in-country seminars a year to feed back evaluation findings. |BIC are 
also starting to feed back lessons learnt to third party countries. For a tourism develop- 
ment project in Iordan, for example, representatives from Thailand attended to explain 
the lessons they had learned. 

11. In outline of the approach is provided in IDS Policy Briefing No. 12, "Participatory Moni- 
toring and Evaluation: Learning From Change" ( 1998). 

12. NGOs and civil society organisations that have been the main pioneers in developing 
PM&E approaches. 

13. See R. Hinton, Briefing Note for the DFID Workshop on an Assessment of Participatory 
Approaches and Methods in Monitoring and Evaluation, London, |uly 2000. 

14. These lessons are drawn from IDS experience (ELDIS, ID21, Bridge, Livelihoods Connect, 
CGAP Microfinance Gateway, etc.) and from others in the development information field. 
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3. Opportunities 


As well as challenges, there are also important opportunities opening up for 
evaluation units. These have the potential for substantially improving the effec- 
tiveness of evaluation feedback, and for integrating lesson learning and account- 
ability into the heart of the development agency planning and thinking. 

3.1 . Taking advantage of the Internet 

For evaluation departments, the introduction of Internet technologies is 
transforming the possibilities for organising, sharing and storing information. 
Although separating the hype from the reality remains a problem, the Internet is 
here to stay, and will be having an ever more important influence on the way 
development agencies communicate and work. From an evaluation perspective it 
offers huge advantages in: 

• Providing secure storage of large amounts of data but in a form that is much 
easier to retrieve than conventional filing systems. 

• Organising and combining information in better and more useful ways. 

• Facilitating the sharing of data, both internally and between agencies, in a 
way that can be managed and controlled far better than paper-based system. 

• Offering a simpler way of providing external audiences with access to infor- 
mation, thereby improving transparency and accountability. 

• Allowing rapid communication across the globe at very low marginal cost. 

Development agencies have made great strides in the past few years in har- 
nessing the new generation of Internet technologies. A total of eleven agencies 
report that "most or all" evaluation reports or summaries are now available on 
their internal intranet, and the same number make this information available on 
their public web sites. Almost all agencies have plans for putting more information 
online, and only a handful have no plans of this kind at present. 

Working out how to use these new technologies effectively has been a steep 
learning curve for most agencies. With new possibilities opening up almost every day, 
one of the problems is that the goalposts keep shifting. Web sites and databases that 
are state-of-the-art when they are introduced can quickly become outdated as the 
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technology moves on and the expectations of users rise. Another problem is that 
of information overload. Systems that are introduced to ease information flows can 
succeed in some respects, but have the simultaneous downside of swamping peo- 
pie with more information than they can deal with. Putting more and more infor- 
mation up on the intranet or web site does not guarantee people will actually use 
it. As the European Bank for Reconstruction and Development put it in their ques- 
tionnaire response, “whatever means one employs to briny the horses to water, ultimately you 
have little leverage ...to make the horses drink''. 

The investment needed in gearing up to make full use of internet-based sys- 
tems is substantial, both in financial and human resources terms. Some evaluation 
departments are better placed to respond than others. Much depends on agency- 
wide capacities and policies. In agencies that are already taking steps to improve 
their information infrastructures and ICT systems, evaluation departments have 
been able to take advantage of this, and upgrade at the same time, in some cases 
taking a leadership role in demonstrating what is possible and pushing for 
improvements. But where the agency as a whole is moving more slowly, and the 
capacity within the evaluation department is limited, it is hard for the evaluation 
department to make progress. Indeed, for some, delays in getting even basic 
material posted on the agency web site or intranet, or in creating links from the 
agency homepage, has been a cause of frustration. 

A comparison of where evaluation units have got to in their web and intranet 
developments is provided in Annex 4. Good practice lessons emerging from the 
first and second generation of web sites and intranets are highlighted in Box 7. 
Some of the key emerging issues are also flagged. 

As the use of the Internet has developed, so too have opportunities for shar- 
ing evaluation information between donor agencies. The DAC Evaluation Reports 
Inventory set up in 1988, and hosted by CIDA and IDRC, was the first major initia- 
tive in this area. The inventory now contains abstracts from slightly over 
5000 reports, contributed by I 7 organisations. Search functions allow the user to 
select reports by donor, country/region, sector, evaluation type, date and keyword. 

Experience with the DAC Inventory has demonstrated both the potential for 
this kind of centralised system and the demands of operating it effectively. Some 
of the difficulties that have been encountered have been related to: 

• Having to depend on inputs from a whole range of donors, some of whom are 
much more diligent than others in keeping their contributions up to date. 

• Differences in cataloguing and indexing approaches among agencies. 

• Conflicting priorities between making documents public and maintaining a 
"private" area where agencies can exchange information on a more confidential 

I 82 basis. 1 
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Box 7. Harnessing the Internet 

Good practice for website and intranet design is developing fast. Some of the key 

lessons that are emerging, and the challenges now on the horizon, are as follows: 

Basic lessons: 

• Clear navigation and signposting is essential, so users do not get lost. 

• Sites need to be kept up-to-date, and be seen to be so, otherwise users 
lose interest. 

• Over-complex designs with a lot of fancy graphics may impress at first, but 
slow down page loading and soon become a turn off. Simplicity and func- 
tionality are more important for regular users. 

• It is often best to organise information in layers, so users see headings and 
summaries first, and can choose where to dig down further. 

• Web sites need active publicity and promotion, otherwise nobody knows 
they are there. 

The cutting edge: 

• As more information is added to websites and intranets, more sophisticated 
databases are needed to handle the volume of material being stored. 

• Better search tools are also required to guide users to the information they 
need. These need to be intuitive and user-friendly, since most people are 
easily deterred by complex looking search screens, or results pages that list 
hundreds of matches. 

• Information needs to be classified and organised in a creative and intelli- 
gent way. It may be better to structure information in a functional way, 
based around how people will actually use it, rather than along standard 
library cataloguing lines. 

• Long lists of documents or web links soon become unreadable, even if the 
material itself is of high quality. Additional text to introduce topic areas, put 
them in context, and highlight what is new, can help guide users into the 
material, and explain its relevance. 

New challenges: 

• Accessibility is being recognised as an increasingly important issue, especially 
for developing country users. New thinking is going into finding compromise 
solutions that offer a high level of functionality but do not depend on having 
the latest equipment or high-speed internet links. Since e-mail is a more 
robust technology, systems that can deliver web documents on request via 
e-mail are one promising route. 
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• The preponderance of older information on the site, and the perceived lack 
of up-to-date material. 

• Not being able to provide direct links through to original source materials, 
which limits the functionality of the system to users (plans are in the pipe- 
line to introduce this facility). 

• Developing a search interface which is powerful but also user friendly. 

It is clear that many agencies value the Inventory, and this is reflected in their 
responses to the efela questionnaire. This showed that 15 agencies contribute 
material on a regular basis. However, it also suggested that it has limitations as a 
practical tool, since only 5 agencies reported that they use the database regularly. 

The DAC Inventory is twelve years old now, and a number of agencies have 
suggested it may be due for a review. Technology has moved on, and with agen- 
cies putting more and more material online on their own sites, fresh options are 
opening up for sharing information in new and potentially more interesting and 
powerful ways. 

A new impetus for reconsidering these options may be provided by the new 
Global Development Gateway initiative being led by the World Bank . 2 This sets out a 
highly ambitious vision for global information sharing. Within the draft Business Plan 
for the Gateway there are outline plans for a section on "Policy Coordination and 
Lessons Learned". The Business Plan has been approved in principle by the 
World Bank Board, and initial seed funding has been released. The intention is to 
flesh out these plans over the next few months, begin discussions with the many 
partners who will need to be involved in making the Gateway idea a success, and 
develop a number of pilot initiatives to demonstrate what is possible. 

3.2. Using evaluation feedback to enhance learning in the organisation 

Development agencies and the evaluation units within them are also being influ- 
enced by the latest generation of organisational management models, in particular: 

• Knowledge Management (KM) 

• Results Based Management (RBM) 

Both are directly concerned with questions of organisational learning and 
accountability, and so overlap very closely with the objectives of evaluation feed- 
back. Since they are usually driven by senior management, and require top level 
support to get off the ground, they provide an important opportunity for evaluation 
units to increase their profile and significance within the organisation, cross-fertilise 
ideas, and build evaluation feedback into broader, agency-wide, processes. 

Most of this new thinking originated in the corporate sector, particularly in 
North America. But it is spreading fast to public sector organisations, and is begin- 
I 84 ning to be adopted by aid agencies . 3 It has spawned a whole new terminology and 
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sub-industry. Although some find the jargon off-putting, and regard it as “old wine 
in new bottles”, these new management trends can be a powerful agent of change . 

Results Based Management is already a part of the DAC agenda. The recent 
re view highlighted twelve main lesson-learning areas and underlined high levels 
of synergy between RBM thinking and improved evaluation feedback. 4 Indeed it is 
difficult to achieve the former without the latter. The review is due to be finalised by 
November 2001. There is thus a good opportunity to ensure that ongoing work in 
these two areas is coordinated in order to optimise progress in both dimensions. 

The broad principles behind some of the new organisational learning models 
are summarised in Box 8. Rather than being an end-point in a linear process, eval- 
uation work, including evaluation feedback, is at the heart of the organisational 
learning cycle. Evaluation departments do not have a monopoly on knowledge within 
an organisation, however through using evaluation feedback positively they can have 
a significant impact. In particular they can play a role in acquiring knowledge through 
evaluation work, and then disseminating it through evaluation feedback; this knowledge 
can then be utilised across the organisation more widely. 

The private sector has been grappling with knowledge management issues 
long enough for a substantial body of literature and lesson learning to have built 
up . A recent survey of 43 1 corporate organisations 5 found that senior managers 
understand that an organisation’s knowledge depends most on its people, but 
that organisations do not have the emphasis right so far, with the split between 
people, process and technology still needing to be reoriented in favour of people. 
The biggest obstacles to change that were identified were cultural and behavioural. 

Within the development assistance community, the World Bank was one of the 
first agencies to embrace the knowledge management agenda. 6 It has been pioneer- 
ing new ways of sharing information and knowledge, though it faces particular chal- 
lenges being such a large and complex organisation. KM has had the advantage in 
the Bank of having top-level backing, and a substantial investment of resources. 
Much has been learnt in the past two years, both at a general level, and in relation 
to evaluation feedback specifically. Some ofthese lessons are summarised in Box9. 

Other agencies are now looking seriously at knowledge management and are at 
various stages in the development of new approaches and systems. 7 Forexample: 

• GTZ has adopted a system of “vertical teams”, 8 for example, which are con- 
ceived along similar lines to the World Bank’s “thematic groups”. 

• UNDP has created the SURF System, a series often “Sub-Regional Resource 
Facilities and Communities of Practice”. These are designed to provide sup- 
port to country offices, encourage networking between UNDP staff and part- 
ners, as well as to identify, document and disseminate best practices in 
UNDP focus areas. 
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Box 8. Evaluation as a part of an Organisational Learning System 

Far from being a discrete operation, 
evaluation can be seen as being at the heart 
of the organisational learning process. 

Developing the “virtuous circle” of effective 
organisational learning demands that evalu- 
ation departments serve to make system 
connections between programme effective- 
ness and organisational effectiveness. 

The process of developing mechanisms 
for effective learning can be seen as a chal- 
lenge that has three interrelated strands: 

• acquisition 

• dissemination 

• utilisation 

This interdependence implies that changes in one part affect other parts and 
call fora response throughout the whole system. 

Acquisition 

Evaluation has to be based on systematic data collection for the improve- 
ment and generation of knowledge about effectiveness. Clear, understandable 
targets which are shared throughout the organisation are the best basis for 
managing-for-re suits. 

Acquisition can be improved by 'making the scanning imperative' - the ability 
to look outside ones own department - a part of the knowledge acquisition culture. 

Dissemination 

Underlying a dissemination strategy is the need to establish the nature of the 
information sharing culture. These include; the degree of openness with which 
issues and problems can be shared and conflict accepted, formal versus informal 
methods and their different role, function and effectiveness, and individual versus 
team orientated objectives for dissemination. 

Utilisation 

The process of evaluation is a highly dynamic, interactive one requiring 
engagement by all parties. This process demands effective facilitation and an 
organisational response. Establishing actionable “lessons learnt" is a key require- 
ment, but this cannot be achieved without feeding this process into leadership 
development. Changes in methods and strategy therefore need to arise from inte- 
grated linkages with the evaluation process. 
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Box 8. Evaluation as a part of ail Organisational Learning System (amt.) 

A key conclusion: shared understandings emerge as evaluation logic 
pushes the senders of messages to be as specific as possible, and challenges 
listeners to reflect on and feed back to senders what they think they have 
heard. 

Systems of interdependent parts imply that changes in one part affect other 
parts and call for adaptation through the whole system. Each part will have an 
effect on the organisation surrounding them, push against boundaries, arrange- 
ments, rules, procedure and attitude. Operating at the organisational level may 
increase the possibility of having impact by being able to deal directly with those 
who have the power to make changes. 


* This bax dtm\s on t\ 10 main sources: Patton (1999), “Organisational Development and Evaluation”; 
The Canadian Journal of Program Evaluation, Special Issue pp. 93-113; and MIT (1994), 'Organisations as 
Learning Systems - a learning resource guide”, The International Consortium for Executive Develop- 
ment Research. 


- SIDAhas a system of mentoring as a mechanism for the exchange of knowl- 
edge (which is often held tacitly). Senior staff have a mentor outside the 
organisation and middle and junior staff are encouraged to have mentors 
within SIDA Mentoring is seen as a useful way of exchanging learning, work- 
ing across traditional vertical reporting lines or sectoral divisions. 

• DFID has developed a computer-based “Performance Reporting Informa- 
tion System for Management” (PRISM). This combines basic project man- 
agement information with qualitative information on the nature and 
objectives of each project. A series of “markers” are used to tag projects. 
This allows analyses to be easily extracted showing breakdowns of spend- 
ing against different poverty objectives, trends over time, the geographic 
spread of funding, and other information. 

• As part of CIDA’s knowledge management approach, efforts are been put 
into creating and managing dedicated “Extranets”. These are accessible 
over the internet using a password, and are being used to facilitate dia- 
logue between communities of practice within the organisation, and with 
outside partners. 

• In USAID, the central evaluation unit is part of a broader-based Centre for 
Development Information and Evaluation (CDIE). This serves as the organi- 
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Box 9. Knowledge Management: lessons from the World Bank 

General lessons 

• Establishing "Communities of Practice 1 ' around particular themes has proved 
to be a good way of focusing information exchange and creating crosscutting 
networks. There are over 100 "thematic groups’’ in the World Bank at present, 
each drawing in members from across the organisation. 

• Web sites and other online information channels can be useful tools, but they 
are only part of the picture. People-to-people networks have proved to be much 
more important, but these only work well when there is a real commitment and 
shared interest among the participants. 

• Help desks and advisory services, staffed by information specialists, can 
play a very useful role as a focal point for answering queries and coordinat- 
ing information. 

• Developing trust between individuals, so people feel safe to share problems 
and queries, is the key to successful knowledge networks. 

• Creating space for learning and knowledge sharing is essential, otherwise 
people are too busy to contribute. Time for KM and training activities is now 
written into job definitions at the Bank. 

• Incentive structures are important, otherwise KM activities tend to slip down 
priority lists. But introducing direct financial incentives is not the best route. 
The respect of colleagues and the desire to "do the job better” are more 
effective inducements. 

• Simple, compelling stories are often the best way of getting new concepts 
across, and winning people over to new ways of thinking. 

Lessons for evaluation departments 

• By being part of specific thematic groups, evaluation staff can develop infor- 
mal links with colleagues in operational and policy departments, and close 
the Feedback Gap. 

• Breaking down the "them and us" barriers that separate evaluation staff from 
other colleagues takes time, but is essential if healthy learning networks are 
to be established. Methodology workshops can be helpful here in "demystify- 
ing” the evaluation process. 

• Evaluation units can help to build their credibility and perceived usefulness if 
they develop their capacity to respond to specific information demands from 
colleagues, rather than relying on "supply driven take the medicine” approach 
to getting information out. 

• Evaluation reports may contain a wealth of relevant lessons, but by them- 
selves be of limited use to operational staff because they are too numerous to 
plough through, or are seen as being too narrow or context specific. Using a 
specially created lessons database, OED is developing a capacity to sift 
through back-files to filter out key lessons, and present them in easy-to-read 
summaries, in response to specific requests. 


* Based on interview with Patrick Grasso, Operations Evaluation Department, World Bank, and with 
Stephen Denning, Chair of the World Bank Knowledge Management Board. 
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sation's institutional memory and has become the focal point for knowledge 
management within the agency. As well as evaluation information, CDIE 
maintains extensive databases of economic and social data on partner 
countries. This is only available on the intranet at present, though discus- 
sions are underway to see if part or all of this can be put on the public 
website. 

On a more modest scale, knowledge-sharing networks of various kinds are 
also proliferating, both within agencies and between them. Networks can form and 
un-form according to need. Recently DFID set up a cluster of networks around the 
process of drafting its new series of Target Strategy Papers. These groups cut 
across traditional country and sector boundaries. At a more formal level, the Eval- 
uation Co-ordination Group provides a forum where multilateral development 
banks can meet to share information, learn lessons and discuss procedures. 9 The 
DAC Working Groups are also a prime example of knowledge networks - although 
they have not been referred to that way in the past. DAC Peer Reviews also create 
periodic nodes of learning which form and un-form, where the results are then dis- 
seminated to the wider DAC group. 

3.3. Linking in with the International Development Goals 

A third important area where opportunities exist for evaluation units to trans- 
form their role and significance is in relation to the International Development 
Goals (IDGs). The development partnerships strategy was adopted by the DAC 
in 1996. The strategy now has agreed goals with a working set of core indicators for 
measuring progress against those goals. All DAC Members are committed to these 
goals, and are collaborating with each other to measure them. Hitherto, this has 
more usually been at the level of their Statistics Departments with some input 
from Evaluation Units. Regular updates are issued, there is a DAC indicators web- 
site and donors now routinely refer to them in their annual reports. Emphasis to 
date has been on refining the indicators, concern over the quality of in-country 
data, and trying to address aggregation problems. However there is scope for 
evaluation units to harness the commitment to the IDGs by their agencies and 
governments, in order to raise the profile of efela within their agencies, and link 
evaluation feedback into the virtuous circle of a learning organisation working to 
improve its development policy and effectiveness. 

The recent DAC Review of Results Based Management observes: 

"... considerable progress towards performance harmonisation has already 
taken place among donors on another front. Broad sector development 
(impact level’) goals and indicators for measuring progress at the country and 
global levels have already been adopted by the donor community in the 
shared strategy, Shaping the 21th Century ■. The Role of Development Co-operation. The 
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DAC Working Party on Statistics has been at the forefront of the effort to iden- 
tify and seek agreement on a common or core set of indicators for measuring 
progress vis-a-vis the shared goals. A number of donor agencies have linked 
their own strategic framework goals and indicators to these international 
efforts. An even more ambitious form of harmonisation among donors and 
partners might be envisioned for the future. Donor agencies' performance 
measurement/management systems could be co-ordinated and tailored at 
the country programme level in a way that lessens the burden on partners 
capacities, builds ownership and enhances development of RBM systems 
within partner countries." 

The Review is quoted extensively as it links so explicitly with the efela 
agenda. The DAC Goals offer a significant opportunity for Members to accelerate 
their strategic thinking, moving towards a virtuous circle, rather than evaluation 
feedback being seen as the end of the line. Monitoring and evaluation systems, 
including feedback measures, could be seen as a part of the solution to the prob- 
lem of ensuring that institutional level learning can be systemised, shortening the 
feedback loop, rather than as a way to measure outcomes in order to justify prior 
investments. 

The IDG initiative could be an important step towards a more systematic 
approach to assessing the impact of donor’s programmes and harmonising impact 
assessment. As donors increasingly move away from project-by-project assistance 
and towards sector wide approaches and medium term expenditure frameworks, 
the demand is growing for greater donor and recipient co-ordination in improving 
the performance monitoring systems that track results. It is already accepted in 
principle that donors need a common systematic rating to allow comparisons of 
results. 10 It is also essential in order not to hinder the development and operation 
of evaluation capacity of partner countries through excessive and conflicting donor 
information requirements. Donors are now just beginning to use the IDGs as a part 
of the top-down performance assessment process. DFID, for example, has drawn 
on the IDGs for its Public Service Agreement with the Treasury (the UK Ministry of 
Finance) and for its more detailed Service Delivery Agreement. Importantly, 
reporting against these goals will (at least in theory) determine access to future 
levels of resources. 

Working from the bottom up, from project to programme and sector delivery, 
donors are also beginning to look at ways to try to measure effectiveness at the 
country level. This process will be accelerated with the advent of the newly agreed 
IFI Poverty Reduction Strategy Paper (PRSP) instrument. The first wave of these is 
planned to be in place by the end of 2000. The PRSPs are meant to be country 
driven, results oriented, comprehensive, long-term in approach and based on 
partnership. In other words, they exactly match the model agenda which evalua- 
tion departments have been trying to promote and follow. Some recipient coun- 
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tries are already pursuing this agenda independently. Uganda announced its own 
poverty reduction strategy last year, for instance, opting to set targets and report 
against the IDGs. There is thus already a high level of ownership in country, so par- 
ticipation and co-ordination problems should be much reduced. 

So whether the methodological approach is from the bottom-up, or the top- 
down, it appears that there is an opportunity for synergy that evaluation departments 
are ideally placed to capitalise on, should they wish to. Evaluation feedback will not 
be the end of a process but part of an ongoing system of reporting against the 
goals and related indicators. If this is linked to access to future resources, there 
will be a built-in vested interest on the part of all parties to generate, obtain and 
use this information. The incentive structures will be systemic. Adopting from 
Charles Leadbeater's analysis of the role of companies in Silicon Valley, 11 evalua- 
tion departments could become the "core junction boxes” for agencies, through 
which information flows in and is generated out. By 2015 if the IDGs are worked 
towards and measured then evaluation feedback for internal and external learning 
and accountability could be part of the same inclusive process. 

3.4. Looking ahead 

The prospects for evaluation feedback can be looked at in a number of ways. 
The external pressures for greater accountability, including measuring and meeting 
the IDGs, could be perceived as a strength, improving the negotiating position of 
evaluation units within agencies. A recent report by the European Centre for 
Development Policy Management talks of "making evaluation the linchpin of the 
new management system". 12 Evaluation feedback would play a key role in this. 

At the same time, current levels of aid "fatigue” or perhaps the misapprehen- 
sion by the public that aid is only associated with famine, flood, war or other 
disaster, could mean that full reporting of aid agency experience may be inter- 
preted as over exposure or weakness. To help counter this tendency, lapan has 
eleven teaching centres nation-wide with a local community outreach and public 
education mandate. DAN1DA and Japan have made evaluation feedback videos 
aimed at a public audience, while DFID has had a "development bus” travelling 
around the UK, and France has a new consultation group including 60 members of 
civil society, academics, industry and commerce. 

There is also the threat that evaluation feedback can be misused or taken out 
of context. The recent World Bank experience with the Meltzer Commission is a 
possible example of this, where evaluation findings may have been taken out of 
context. 13 USAID may also be said to have suffered at times from extreme cases of 
"micro-management" by the US Congress, 14 though has recently been praised for 
their honest and straightforward Annual Performance Reports, and for their open 
policies on disclosure of evaluation reports. 
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The overall conclusion emerging from this study, however, is that the efela 
agenda provides an exciting and important opportunity for all development agen- 
cies. There will no doubt be a range of views as to how possible or practical some 
of the opportunities outlined in this report may be, or how applicable they are to 
the agendas of the wide range of agencies concerned. At a recent meeting of the 
DAC Working Party on Aid Evaluation four areas of pressure for developing donor- 
wide evaluation policies were identified: 

• The organisational demand to say something positive about progress 
towards poverty reduction targets, both internal and external (IDGs) 

• Financial pressure 

• Political pressure 

• Administrative pressure 

Improving efela can be seen as a win-win option addressing all four of these 
areas. The forthcoming workshop in Tokyo will provide an opportunity for as many 
stakeholders as possible to discuss how this might best be achieved. 
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Notes 


1 . The advantage of having a "closed space" for donor plans and "non-public" documents 
has led to an agreement to establish a closed access web-site by September 2000, 
available to contributors only. 

2. See www.worldbank.org/gateway . 

3. A summary of how this new thinking is affecting the development agencies is provided 
in the recent DAC report, “A Comparison of Management Systems for Development 
co-operation in OECD/DAC Members", DCD(99)6. 

4. Results Based Management in the Development co-operation Agencies: A Review of 
Experience DCD/DAC/EV(2000)3 and summarised in DCD(99)6. 

5. A |oint Survey by Business Intelligence and The Ernst and Young Center for Business 
Innovation, 2000 (see www. businessinnovation.ey.com/research/knowle/survey ) . 

6. The term "knowledge management" has been phased out by the World Bank in favour 
of the more organic term, "knowledge sharing". 

7. A more detailed picture of current thinking is available in the background documenta- 
tion prepared for two recent Workshops on "Knowledge Management: Implications and 
Applications for Development Organizations”, held in Washington and Brighton, organ- 
ised by Bellanet (see www.bellanet.org/km ). 

8. As the GTZ interviewee commented, "the person is the carrier of the know-how, the 
question is how to liase with other people to capture the know-how”. 

9. The ECG has recently produced a best practice paper for lessons learned in dissemination 
"Good Practice in Lessons Learned - Dissemination and Application”, Wolfgang Gruber, 

EBRD May 2000, submitted to the MDB ECG. 

10. Donecker, ). and Green, M. (1998), Impact Assessment in Multilateral Development 
Institutions, DFID. 

1 I . Charles Leadbeater (2000), "Living on Thin Air' - The New Economy", Penguin. 

12. "Assessing Trends in EC Development Policy - An Independent Review of the EC's 
External Aid Reform Process", ECDPM Discussion Paper No. 16, May 2000. 

13. "Development Effectiveness at the World Bank: What is the score?" OED Reach No. 14, 
spring 2000. 

14. A Comparison of Management Systems for Development Cooperation in OECD/DAC 

Members. DCD(99)6. 93 I 
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Annex 1 

Conclusions from Previous Evaluation Feedback Studies 


1 . 1990 DAC Review of Donors’ Systems for Feedback on Aid Evaluation 
Key observations and lessons learned were that: 

• Feedback is a dynamic and continual part of the evaluation process 

• A multidimensional approach to feedback is required, including refining and targeting 

• A systemic approach to feedback is required 

• Effective dissemination mechanisms are needed eg. seminars, synthesis reports 

• Effective integration of evaluation into the planning process is needed 

• Internal processes need to be strengthened by formal and informal means 

• Support from senior management is required 

• With donor agency decentralisation effective feedback is even more important 

• How to measure success? - the growing importance of impact assessment 

• Involving recipient countries in the feedback loop is important, joint evaluations 

• Evaluation information exchange between donors can be valuable 

2. 1991 Review of the Effectiveness of Feedback Mechanisms in Member States and the 
EEC Commission 

Recommendations were grouped under two headings, and stressed: 

Feedback materials 

• More focus on sustainability 

• Getting the right topics at the right time to the right people 

• The need for clues for success as well as for failure 

• The need for single sheet summaries with a compatible classification and ring-binder 
system 

• Improved beneficiary participation 

• Human-interest media-friendly take on evaluation findings 

• Deriving broader lessons 

• Annual Reports could include a response to recommendations in evaluations 
Feedback mechanisms 

• Developing informal feedback mechanisms 

• Developing an effective feedback chain involving senior management 

• Action oriented feedback - digested so that senior management can use it 

• Using the Integrated Approach and the Logical Framework 

• Building a Corporate Memory, using the DAC Inventory 

• Using evaluation reports for staff training 

• A policy of openness, selectively where necessary 
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Annex 2 

Background Questionnaire and Results 


Replies to the EFELA background questionnaire were received from the evaluation 
departments of the following agencies: 


Australia (AusAid), 

Switzerland (SDC) 

Austria (Federal Ministry for Foreign Affairs) 

UK (DFID) 

Canada (CIDA) 

USAID 

Denmark (DANIDA) 

Finland (Department for International 

France (Agence Fran^aise de Developpement - 
AFD, Ministere des Affaires Etrangeres - MAE) 

Development Co-operation) 

Germany (GTZ, BMZ, KfW) 

Ireland (Department of Foreign Affairs) 

Japan (J1CA, Ministry of Foreign Affairs, JBIC) 

Luxembourg (Ministry of Foreign Affairs) 

EC 

Netherlands (Ministry of Foreign Affairs) 

UNDP 

New Zealand (Ministry of Foreign Affairs and Trade (African Development Bank (AfDB) 

Norway (Ministry of Foreign Affairs) 

Asian Development Bank (ADB) 

Portugal (Institute of Portuguese Cooperation) 

EBRD 

Sweden (SIDA) 

World Bank-OED 


Notes on the data: 


The following aggregate results reflect the responses of all 29 agencies. However, while most 
of the questions were answered, at least in part, by all agencies, variations occurred within some 
of individual questions. Multi-codes answers were given by some agencies and some sections 
were left blank. The bases given for each question refers to all those who replies to at least part 
of the whole question. Where relevant, an explanation of these variations has been included. 

B I . Does your agency have a formal strategy for dissemination and feedback 
of evaluation results? 


14 Yes - we have a formal strategy* 

15 Some elements exist, but not a formal strategy 
3 We are currently working on a strategy 

0 No formal strategy or guidelines exist at present 


(n = 29) 

* Multi-code responses 

Two members responded that they have “some elements exist " and “we are currently working on a strategy". I 
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Selection of Hey comments and issues raised: 

• As there are a number of different types of evaluation (sectoral, geographic, partner- 
ship) there is no one system which will work in any situation (France MAE). 

• Policy reactions, written by the Ministry will be fed back to the relevant stakeholders, 
but this is not the responsibility of the evaluation department (Netherlands) 

• The plans for dissemination and the response to evaluations should be formally doc- 
umented and are part of the ordinary planning process (Sweden) 

• Strategy to date has focused more on improving the presentation of reports (UK) 

• An evaluation committee is formed, including heads of operational departments, pol- 
icy studies department and management control department - and implementing the 
resolutions is the responsibility of the evaluation department (France ADF). 

• Strategy covers feedback of lessons in the project cycle, but not external dissemina- 
tion (]apan-)ICA) 

• Guidelines include the requirement to make findings available centrally through the 
development information centre (US) 

• Informal guidelines in practice and the need to develop a formal strategy is recognised 
(New Zealand, Canada) 

• There exists no single document, but (the ex-post evaluation methods) is a well- 
known procedure with long institutional experience (Germany-KfW). 

B2. Which target audiences are seen as the main priority for evaluation feedback? 


Not 

important 

Fairly 

important 

Important 

Very 

important 

Average 

0 

5 

7 

18 

Policy makers and senior management 

3.4 

0 

7 

7 

19 

Technical staff involved in project planning 
and management 

3.1 

1 1 1 

10 

10 

Partner countries, and stakeholders within 
partner countries 

2.6 

6 

9 

9 

3 

The media and the general public 

2.0 


(n = 28) 

Sweden's response could not be included as they did not rank them, they did however indicate that all the audi- 
ences mentioned were important. 

Selection of key comments and issues raised: 

• Dependent on the objective of the evaluation (Sweden, Switzerland, US) 

• Media and public are beginning to become more important (Canada) 

• In reality the dissemination mechanisms in Germany are stronger than in our partner 
countries (Germany-KfW) 

• With a limited budget and staff, feedback to partner countries has not happened to a 
satisfactory level (Japan - MoFA) 
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• Due to recession in the lapanese economy the general public are more critical of ODA 
spending (]apan-(ICA) 

• "Action orientated evaluation” means that policy and technical staff are the first target, 
public later (African Development Bank) 

• We see communication to the public as important, but at present we don't do this as 
well as we would like (New Zealand) 

B3. Does the evaluation department have primary responsibility for coordinating 
the dissemination and feedback of evaluation results for learning purposes 
within the organisation? 


18 Yes - the evaluation department has prime responsibility 
12 Responsibility is shared with ( please specify*): 

I No - it is the responsibility of ( please specify*): 


(n = 28) 

‘Multi-code responses 

Australia, the EC and 'World Banfe-OED responded that they have the evaluation department has responsi- 
bility, but it is also shared. 

Luxembourg did not respond to this question, on the grounds that “The responsibility lies with the Directorate 

for Development Cooperation, Luxembourg has no evaluation department". 

Selection of key comments and issues raised: 

• The evaluation department writes an information bulletin for each evaluation, the 
information department writes a press release. In both cases, the text is discussed and 
adopted in mutual co-operation (Netherlands). 

• Evaluation is seen as a participatory process of learning, taking place at various levels 
of management. In this context, responsibilities are shared, with the Evaluation 
Department co-ordinating activities and, at times, taking the lead role (Austria). 

• (The evaluation department only has responsibility in the short term). The long-term 
responsibility lies with the international development information centre, which 
includes the Corporate Memory Unit, and the initiatives of individuals (Canada). 

• (Responsibility is) Shared with technical department - external internal split - 
Germany-BMZ auditor, EVD is the internal (Germany-GTZ). 

• The Bank's Knowledge Management system incorporates evaluation results through 
thematic groups as well (World Bank-OED). 

B4. Does the evaluation department have primary responsibility for coordinating 

the dissemination and feedback of evaluation results for external accountability purposes 

(e.g. to parliament, the general public, stakeholders in recipient country, etc.)? 


9 Yes - the evaluation department has prime responsibility 
16 Responsibility is shared with [please specify*): 

4 No -it is the responsibility of [please specify*): 

(II = 29) 97 I 
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Selection of key comments and issues raised 

• In many instances (not a standard procedure!) the publication of an evaluation report 
is combined with a press conference in which the evaluation department provides any 
background information to the press (Netherlands). 

• It is shared with political departments and with programming unit but not in a system- 
atic way (Portugal) 

• The report of the ex-post evaluation is sent to the Ministry (Germany-BMZ) and in very 
condensed form to the partner organisations with prime responsibility of the country 
department, whereas the biannual report "Results of Financial Cooperation" for the 
public is produced mainly by the sector policy department (Germany-KfW) 

• Responsibility for the dissemination of evaluation reports to third parties is shared 
between operational departments, the policy and studies department and the evaluation 
unit, depending on the status or characteristics of the targeted public. (France - AFD). 

• We get assistance from the Public Relations Division in disseminating evaluation 
results through ]apan-]ICA Homepage ()apan-|ICA). 

• (Main, domestic responsibility rests with CDIE). US country field offices have respon- 
sibility for coordinating dissemination and feedback of performance and evaluation 
results to recipient country partners [e g., governments, NGOs, etc. involved in the 
activity as well as other donors) (US). 

• Timing and modus is coordinated with the senior management and press office, the 
contents/message is formulated by Evaluation Secretariat (Denmark). 

B5. Does the evaluation department have staff with specific responsibility 

for dissemination and feedback work 


7 Yes - we have nominated staff working in the department 
I Yes - we use consultant(s) to coordinate dissemination and feedback work 
3 No - but we enlist the help of specialist staff working elsewhere in the organisation 
23 No - dissemination and feedback work is incorporated into genera] staff responsibilities 
1 Other arrangements ( please outline ): 


Multi-code (n = 29) 


Selection of key comments and issues raised: 

• Head of the mission and consultants involved (France MAE). 

• Working to establish "knowledge centres" to provide support to operational depart- 
ments, to become an instrument of dissemination of materials from other donors, 
analyse the quality of decentralised evaluation work (Netherlands). 

• Professional editor and information department (UK). 

• Sector economists, informal sector networks engineering department, internal auditor, 
management committees (Germany-KfW). 

• Performance Information and Assessment Section and the Office of Program Review 
and Evaluations - managing the lessons learnt database (Australia). 

• An assistant working in the evaluation department (France AfD). 

• Hired consultant made a system for evaluation database (|apan-|ICA). 

• CDIE has staff who manage contractors (a "clearing house”) that are directly responsible 
for electronic and hardcopy dissemination of Agency-wide performance and evaluation 
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reports to internal and external audiences. On request, these contractors also identify 
and analyse development experience from both internal and external sources and 
disseminate these to US staff and on a more limited basis to partners (US). 

• We enlist help of the Training and Education Department (Denmark). 

B6. Does the evaluation department have a specific budget allocated for dissemination 
and feedback work? 


7 Yes 
20 No 

(n = 28) 

Luxembourg did not reply as - "financial resources are made available from the evaluation/audit budget 
if the need arises." 

France MAE said that there was no specific budget, but "we can consider that the budget 
allocated for dissemination and feedback work is about 10%". 


Of the agencies that responded yes - the following ( approximate ) amounts were quoted: 

UK- 

4% 

US - 

10% 

Germany - 

10% 

Japan-)BIC - 

10% 

]apan-MoFA - 

12% 

World Bank-OED - 

10% 


Selection of key comments and issues raised: 

• All cost of feedback (such as disseminating evaluation reports, seminars, lectures etc.) 
are borne by the activity budget (part of the evaluation budget) (Netherlands). 

• The budget for feedback work is usually calculated separately and is a share of the 
budget for evaluation of a programme (Sweden). 

• As the evaluation Department is free to decide on how to utilise its budget, funds could 
be allocated for this purpose. However, so far no funds have been specifically ear- 
marked (Austria) 

• 10% includes the cost of publishing/translating the evaluation reports and feedback 
seminars in partner countries (|apan-|BIC). 

• The budget is used for publishing annual report, videos, leaflets etc. (Japan-MoFA). 

• We don’t distinguish the budget for disseminating and feedback from others in our 
budget planning (]apan-)ICA). 

• There is no specific budget allocated but is part of the department's activities which 
absorbs approximately 10% of annual staff time (African Development Bank). 

Cl . How important are evaluation reports as a dissemination vehicle? 


15 They are the main vehicle for disseminating evaluation results 
1 2 They are an important vehicle, but not the main one 

2 They are not a key dissemination vehicle, other mechanisms are more important 

(H = 29) 
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Selection of key comments and issues raised: 

• The final report is certainly an important tool as it gives credit to the quality of the work 
but the dissemination process has to start from the beginning, i.e. during the prepara- 
tion of terms of reference: how the evaluation is going to be used is a question often 
put forward in the first meeting of the steering committee when participants are given 
the first draft of terms of reference (France-MAE). 

• The reports are the basis and prime information source and the reports are widely distrib- 
uted within the Bank, However, this is not good enough. The evaluation function employs 
various mechanisms to bring the messages across, including: a) PED's contribution to the 
Bank's Annual Report; b) PED’s Annual Evaluation Overview Report (AEOR); c) on-line 
availability (for Banking staff) of PED’s Lessons-Learned data base; d) regular work shops 
with Bank-approval authorities, individual banking teams (on cross-sector or cross-country 
issues), new bank staff joining EBRD and other mechanisms (EBRD) 

• Although the reports are considered as the main vehicle for feedback, it is the experi- 
ence of the evaluation department that feedback during the evaluation process is 
equally important. The evaluation department normally establishes so-called refer- 
ence groups providing advice during the evaluation process (these groups have 
included officials from the partner country) (Netherlands). 

• (Reports are the main vehicle for dissemination outside Commission); but internally, 
participation in evaluation processes ensures dissemination and feedback even 
before final reports are completed (EC). 

• The involvement of primary stakeholders in the evaluation process is an important 
(maybe the most important) learning and dissemination mechanism. The dissemina- 
tion of reports are mainly important for more audiences further away, e.g. researchers, 
consultants, media and the general public. To reach these audiences it is often neces- 
sary to summarise the reports in Newsletters, etc. (Sweden). 

• Apart of our ex-post evaluation reports (which are also disseminated in a summarised 
report to the public: "Results of Financial Cooperation") we are also doing cross-section 
analysis of ongoing and evaluated projects and programmes, in specific themes or sectors. 
Best practice cases are disseminated monthly via our intranet (Germany-KfW). 

• While considered extremely important for internal distribution, Australia also dissem- 
inates its evaluation findings through seminars, briefings, publications, the intranet, 
the LLDB, the Activity Management System and the Australia website, in the longer 
term, the reports are arguably the most important dissemination vehicles particularly 
in view of their publication on the internet (Australia). 


C2. What efforts are made to provide short summaries of evaluation reports? 


0 Some evaluation reports contain summaries, but this is not a required feature 
25 Most/all evaluation reports now contain a short summary 
6 Separate summaries* are produced for selected evaluation reports 
10 Separate summaries are produced for most/all evaluation reports 


Multi-code response (italics denotes agencies that have replies with more than one response) 


(n = 29) 
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Selection of key comments and issues raised: 

• All evaluation reports have summaries which are distributed to all banking staff. For 
Board members "cleaned" (from commercially confidential material) versions are pro- 
duce d (EBRD) 

• Evaluations of wider interest are summarised and presented in a regular Newsletter, 
SIDA Evaluations Newsletter (Sweden). 

• We believe that a multi-layered approach is required to ensure awareness at all levels 

(Canada). 

• (Separate summaries - of all evaluation reports). Only in ]apanese (3-5 pages with 
photos and a map) (|apan-)BIC). 

• We do not produce short summaries for our regular ex-post evaluations. In fact, our 
project evaluation reports are confidential and sent only to the government. Following 
the regulations of FC we are not allowed to publish them. In the case of evaluations for 
Germany-BMZ commissioned by Germany-KfW we provide for several versions of 
short summaries (for Germany-BMZ homepage, DAC, partner country in respective 
languages). We also produce main reports and summaries of special evaluation mis- 
sions with the objective of publication (Germany-KfW) 

• Separate summaries together with proposals for measures to be taken on the basis 
of the findings and recommendations presented in the Report are submitted to 
the Management Board (Finland). 

• Apart from summary sheets (4-6 pages) broader Danish summaries (approx. 30 pages) 
are prepared for selected evaluations (Denmark) 

C3. Is dissemination and feedback of interim results during the evaluation process seen 

as a priority? 


15 Yes - the main mechanisms used are ( please specify): 
7 No - not at present 
7 it is in some cases (please specify ): 


(« = 28) 

France-MAE did not respond to this question. 

Ireland responded to the question as “no" and “in some cases" (see comments below). 

Selection of key comments and issues raised: 

• Even if it not stated as a priority it often become one under the pressure of decision 
makers (approval of a new phase), and expectations of different stakeholders. The 
steering committee which is put up for each evaluation gives the main and more effi- 
cient impulse for dissemination (France-MAE). 

• At least, this is not a formal requirement. However, de facto , consultations with respon- 
sible banking staff take place during an evaluation process, particularly if it concerns 
more complex, longer so-called Special Studies (EBRD) 

• It is our feeling that the feedback during the evaluation process is a very important 
vehicle for internalising the evaluation results in the organisation (both the ministry as 
well as at the field level (Netherlands). 
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• Where major evaluation involve a desk-top and field phase, the result of the desk - 
top may be presented in the form of an aide-memoire (Ireland) 

• It is in some cases (please specify): e.g. when these interim results give reason to act 
immediately to strengthen certain aspects during the ongoing implementation, or to 
prevent problems indicated in the interim report (Austria). 

• The review of the draft reports probably offer the greatest opportunity for institutional 
learning (Canada). 

• Rather than a "priority", I would say that dissemination of interim results is a normal 
(and compulsory) step in the production of evaluation reports. Within a week after the 
completion of their mission in the field, evaluators must produce a short briefing 
memo that summarises the main findings of the evaluation. Then, they have to pro- 
duce a "draft evaluation report” which is submitted to the Evaluation Committee. 
Finally they produce the "final evaluation report" (France-AFD). 

• Results from CDIE US-wide evaluations are disseminated in draft to selected technical 
and management staff and in some cases to externa! technical experts for comment 
and discussion prior to preparation of final versions. At the country level, interim 
results may be reported orally to US staff and country partners (US). 

C4. Are active steps taken to ensure evaluation reports are fed back to in-country 

partners and stakeholders? 


20 Yes - this happens routinely 

6 It happens in some cases, but not systematically 
2 It happens rarely/never 


France-MAE did not respond to th is question. 


(n = 28) 


| 102 


Selection of key comments and issues raised: 

• So far, evaluation work is primarily seen as an accountability function towards fund 
providers (Bank's shareholders, TC donors) and as a quality management function 
towards bank staff and management. Note should be taken that related experiences 
in other organisations generated rather mixed results in addition of external consulta- 
tion being a rather time consuming process requiring additional administrative 
resources and which would ultimately require a joint evaluation process (EBRD) 

• It depends on the purpose of the evaluation (Sweden). 

• This is going to be an integral part of the evaluation strategy being discussed at the 
moment (Austria) 

• This is an absolute priority (Luxembourg). 

• We need to do better but, we need to remember the additional time requirements 

(Canada) 

• We send draft evaluation reports to in-country partners for comment - but seldom 
receive much by way of written reply or commentary (UK). 

• It depends on the scheme. Every country evaluation is followed by a seminar in recip- 
ient countries. Besides that a limited number of reports are translated and fed back to 
in-country partners ()apan-MoFA). 
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• In certain exceptional cases, e.g. of political sensitivity reports may not be fed back (in 
full) (Finland) 

• This is standard operating procedure for evaluations of country field level activities (US). 

• The feedback to in-country partners and stakeholders most often takes place as work- 
shops/dialogue during or at final stages of the evaluation process. Later feedback is 
supported by embassy/desk officers/specialists (Technical Advisory Service, TSA) 

(Denmark). 

C5. Is your agency Intranet used for disseminating evaluation results? 


2 S ome evaluation reports/summaries are now available on our internal intranet 

1 1 MosI/ all evaluation reports/summaries are now available on our intranet 

12 We have plans to make evaluation reports/summaries available on our intranet in future 

6 We have no plans at present to use our intranet in this way / we do not have an intranet 


(n = 29) 

Australia and France-MAE responded with two of the options. 

Selection of key comments and issues raised: 

• )ust a note of caution, whatever means one employs "to bring the horses to the water", 
ultimately you have little leverage (although more could be done definitely) “to make 
the horses drinking". There is still a long way to go before evaluation work is "demand- 
pulled" rather than “supply-driven” (EBRD) 

• (no intranet) As we do not want to duplicate the DAC inventory (Switzerland). 

• Following a recent review, UK's Intranet is likely to be redesigned with a specific section 
on "performance assessment" which should help to give increased profile to evaluation 
and related activities/information (UK). 

• We have a plan to build a database which will be available (thorough the| Intranet in 
the near future. We already have a web-site in which evaluation summaries are avail- 
able. But still reports are the main vehicle for dissemination (Japan-MoFA). 

• So far our intranet is not used for disseminating evaluations results from regular 
project evaluation (since we evaluate every single project, the number of evaluations 
p.a. amounts to about 100 to 12011. However, results of cross-sectional analysis is avail- 
able. It will be one of the tasks of the new evaluation department to build up a new 
intranet page for disseminating evaluation results (Germany-KfW). 

• Until now, only the resolutions of the Evaluation Committee and the list of project 
evaluation reports are regularly stored on our intranet and updated (France - AfD). 

C6. Is your agency website used for disseminating evaluation results? 


6 Some evaluation reports/summaries are now available on our public web site* 

1 1 Most/all evaluation reports/summaries are now available on our public web site* 

8 We have plans to make evaluation reports/summaries available on our public web site 
5 We have no plans at present to use our web site in this way 


(M = 29) 
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Selection of key comments and issues raised: 

• We are preparing a new web site where summaries, and eventually reports, will be avail- 
able. Contribute regularly to the DAC Evaluation Reports Inventory. Until now summaries 
have been produced only for evaluation undertaken by the Evaluation Department. We 
plan to extend it in the future and to make a global base (France - MAE). 

• Our web site only contains a list of evaluation reports produced since 1998. Plan to make 
one-page summaries shortly available on our site. Availability of sector summary 
reports is also envisaged (France - AfD). 

• We do not yet avail of a page in our ministry's web site. We are presently discussing 
publication of our reports on a special "development oriented” web site (EUFORIC) in 
the Netherlands (Netherlands). 

• There are plans to increase availability on the Internet (Canada). 

• SDC is contributing to CIDA managed database (Switzerland). 

• Following a recent review, DFID s Intranet is likely to be redesigned with a specific sec- 
tion on “performance assessment" which should help to give increased profile to eval- 
uation and related activities/information (UK). 

• Evaluation reports available for public distribution are made available on the website 

(Australia). 

• We are in the process of updating our website, and have discussed making evaluation 
results available on the new site (New Zealand). 

• Under the webpage subtopic of “ Publications/Partners", one may access electronically 
CDIE-conducted evaluation reports dating from 1995, Agency Performance Reports 
dating from 1995 and country-level performance reports (R4s) dating from 1999. 
Included are all evaluations as well as other technical reports, assessments, feasibility 
studies and the like (USA). 

Agency evaluation web sites 


W here the agency has a specific evaluation area of the site, the web address has been shown in bold type. 
Note that live links to these pages are provided from the efela website (www.ids.ac.uk/efela) 


Australia 

www.Australia.gov.au/publications/evaluations 

Austria (Federal Ministry for Foreign Affairs) 


Canada 

w3.acdi-cida.gc.ca 

Denmark 

www.evaluation.dk 

Finland (Dept for International Development 


Co-operation) 


Ireland (Department of Foreign Affairs) 

www.irlgov.ie/iveagfi/irisfiaid/evaluations/evaluation.htm 

Luxembourg (Ministry of Foreign Affairs) 

www.etat.lu/internet.fitml 

Netherlands (Ministry of Foreign Affairs) 

www.dutcfiembassy.bg/netscape/f_netscape.fitm 

New Zealand (Ministry of Foreign Affairs and Trade) 

www.mft.govt. nz/nzoda/adaf 

Norway (Ministry of Foreign Affairs) 

www.norad.no/ 

Portugal (Institute of Portuguese Cooperation) 

Under construction 

Sweden 

www.sida.se 

Switzerland 

www.sdc.admin.cfi 

UK 

www.dfid.gov.uk/public/wfiat/wfiat_frame.html 

US 

www.info.usaid.gov 
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France Ministere des Affaires etrangeres 

Agence frangaise de 
deeveloppement 
Germany Germany-KfW 
Germany-BMZ 
Germany-GTZ 
Japan Japan-JBIC 

Ministry of Foreign Affairs 
Japan-JICA, 

EC 

UNDP 

African Development Bank 
Asian Development Bank 
EBRD 

World Bank -OED 


ww w. di p lorn a tie. fr/coope ra tion/i ndex.html 
www.afd.fr 

www. kfw.de 

www.bmz.de/epolitik/evaluierung/index.html 

www.gtz.de 

www.jbic.go.jp 

www.mofa.go.jp 

www.jica.go.jp 

europa.eu.intlcommlscrleva\uationlindex.htm 
stone. undp.org/undpwebldimalerlfiXesearch.cfm 
www.afdb.org 
oeo.asiandevbank.org 

www.ebrd.com 

www. world bank. org/html/OED/evaluation 


*T hese sites were accessed on tfie 29/06/00. 

C7. What other channels are used by your agency for dissemination and feedback 
of evaluation results, and how important are they? 


Not 

important 

Fairly 

important 

Important 

Very 

important 

(« = 28) 

A verage* 

1 

4 

4 

15 

Annual reports summarising evaluation 
experience* 

3.0 

2 

5 

1 1 

4 

Other summary reports covering particular 
sectors/regions* 

2.3 

3 

7 

6 

3 

Media briefings/press releases* 

1.7 

7 

3 

2 

0 

Videos and other audio-visual outputs 
(please specify ) 

0.7 

6 

0 

2 

7 

Permanent evaluation review committees 

1.5 

3 

6 

6 

12 

One-off staff meetings/workshops to consider 
particular evaluation reports or topics 

3.0 

1 

4 

10 

6 

General staff workshops/seminars 
and training events 

2.3 

2 

5 

7 

7 

Recipient country stakeholder workshops/ 
seminars 

2.3 

1 

4 

10 

10 

Dialogue between evaluation unit and other 
agency staff in response to specific enquiries 

2.9 


* averages are calculated by taking "very important = 4" to "not important = 1 The base of 28 is used, and it 
is assumed that a lack of response means a "not important" ranking. UNDP’s results could not be used, as they 
were not ranked, only ticked. 

Selection of hey comments and issues raised: 

Other mechanisms mentioned: 

• Newsletters, management response. The channels depend on objectives and thus 
audiences of the evaluations. All this should be discussed in the plan for the dissem- 
ination of evaluation reports (Sweden). *°^l 
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• Synthesis studies have become a key vehicle for collating and disseminating evalua- 
tion results. We would like to arrange more in-country stakeholder workshops and 
seminars, which have good potential for dissemination in-country - the low score 
reflects current (limited) practice (UK). 

• Program Quality Committee reviews the broad profiles of all evaluations conducted by 
the Agency (Australia). 

• Feedback to the recipient administrations and companies is also important. However, 
this feedback rarely takes the form of workshops, seminars and such, but rather of 
working meetings between our staff in the field and the staff of the recipient country... 
Dialogue with other evaluation units of development agencies is not formalised nor 
systematic but may occur on the occasion of joint evaluations (France - AFD). 


C8. Are any formal mechanisms in place to make sure that lessons from evaluations 
are taken properly into account? 


1 4 Specific evidence of lesson learning is required in new spending proposals 

20 Senior management are expected to respond to evaluation findings and recommendations 

10 Evaluation unit staff are involved in the review and approval of new spending proposals 

13 Recipient countries are expected to respond to evaluation findings and recommendations 

4 No formal mechanisms exist 
4 Other (please specify): 


(ft = 29) 


Agencies that responded with more than one answer are denoted by italics. 


Selection of key comments and issues raised: 

Other mechanisms mentioned: 

• Meetings of the Portfolio Review Committee (PRC) which commissions evaluation 
work through approval of the evaluation department's work programme, and then 
reviews key studies produced. Some of these studies are referred to the Development 
Policy Committee, which has a stronger focus on policy formulation (UK). 

• The evaluation unit is responsible for the monitoring of the enforcement of the reso- 
lutions of the Evaluation Committee. Departments and country offices concerned by the 
recommendations of the Committee are periodically requested by the unit to indicate 
the progresses made in implementing these recommendations. However, the evalua- 
tion does not participate to the management committees, which examine the new 
projects. The policy and studies department, which is represented in these committees, 
should normally verify that the lessons learned from evaluation are taking into account 
in the design of the project (France-AFD) 

• We have a formal mechanism to check whether the evaluation recommendations have 
been put in place approximately one year after each evaluation. The findings of this 
inquiry are submitted to the State Secretary (Germany-BMZ). 
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C9. What mechanisms/channels are used for inter-agency lesson learning, 

and how frequently are they used? (apart from DAC Working Group channels) 


Rarely/ 

Never 

Occasionally 

Frequently 

Average 

6 

16 

1 

We take part in joint evaluation missions 

1.5 

5 

4 

15 

We contribute material to the DAC Evaluation Reports 
Inventory hosted by Canada 

2.1 

8 

12 

5 

We use material from the DAC Evaluation Reports 
Inventory 

1.7 

12 1 

5 

We have formal arrangements for sharing documents and 
experience with other agencies ( apart from via DAC channels ) 

1.0 

3 

1 1 

1 1 

We share documents and experience informally with other 
agencies 

2.1 

0 

2 

4 

Other ( please specify): 

0.6 


(n = 25) 

^averages are calculated by taking “frequently" = 3, " occasionally " = 2, and " rarely or never” = I. It is 
assumed that a lack of response means this channel is "rarely or never". 

Selection of key comments and issues raised: 

Other mechanisms mentioned: 

• Annual meetings of EU heads of evaluation is a useful forum for sharing information. 
The UK Evaluation Society annual conference is also useful (UK). 

• We take part in working groups of the German evaluation society (DeGEval) with 
NGOs, governmental agencies and scientists, the DAC Working Party on Aid Evalua- 
tion meetings and workshops, publications of World Bank and other donor agencies 

(Germany-KfW). 

• Our problem is that we have rarely made English version of evaluation reports. But 
from this year we are going to make English version of the annual evaluation report 
and summary reports of country-wise and thematic evaluation studies. With this new 
arrangement, we can promote exchanging information on evaluation with other donors 

(|apan-|ICA) 

• The Bank is member of the Multilateral Development Banks' Evaluation Cooperation 
Group (ECG), established in 1996. ECG meets twice a year (AfDB). 
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D 1 . The 1 998 Review of DAC Principles found that the adequacy of dissemination 

practices was a major concern. For your agency, what do you see as the areas most 
in need of improvement? 


Not 

important 

Fairly 

important 

Important 

Very 

important 


Average* 

3 

8 

8 

8 

The type/format of information produced 
needs rethinking, with more emphasis on 
clarity and readability 

2.8 

4 

8 

9 

6 

More care is needed in tailoring evaluation 
|outputs to specific target groups 

2.6 

7 

8 

6 

4 

'Evaluation results need to be distributed 
|much more widely 

2.3 

1 

5 

10 

9 

Steps are needed to make evaluation 
feedback more timely 

3.1 

3 

9 

10 

3 

Steps are needed to encourage lesson- 
sharing and feedback during the evaluation 
process 

2.5 

0 

2 

6 

17 

Steps are needed to institutionalise the 
process of learning lessons within the agency, 
so evaluation feedback is taken more 
|seriously 

3.6 

1 

5 

7 

12 

Steps are needed to encourage better sharing 
of evaluation results with stakeholders 
in recipient countries 

3.2 

0 

7 

14 

4 

]Steps are needed to encourage more inter- 
agency lesson learning 

2.9 


(n = 29) 

•averages are calculated by taking 4 = very important 3 = important 2 = fairly important I = not important. 
A lack of response is interpreted as implying a ranking of “not important' 1 . 


Selection of key comments and issues raised: 

• An evaluation which is not bringing any learning could be worse than no evaluation at 
all, but many evaluations have some impact even before they give any conclusions, 
just being participative (France-MAE). 

• The most important issues here are to encourage harmonisation and collaboration 
among donors and greater capacity and ownership in partner countries. The high 
scores here reflect aspirations not practice (UK). 

• Evaluation on "impacts" of the project should be weighted more, since the public are 
interested most in this aspect ((apan-)BIC). 

• We are considering the need of a flexible strategy for broader and "ongoing” dissemi- 
nation of specific issues from evaluations, such as gender questions, role of private 
business for various stakeholders and target groups through tailor-made seminars/ 
workshops, handouts etc. (Denmark). 

• In the specific context of the Agency’s projects, Australia is working to tighten the var- 
ious levels of the relationship between evaluations and lessons so that they can be made 
more relevant to the different aspects of the activity cycle and impact positively on the 
quality of projects. It is also examining the structure and function of its administrative units 
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to break down the "silo effect" and achieve better outcomes through knowledge- 
based synergies (Australia). 


D2. With the changing context of development assistance, what do you see as the most 
important emerging trends relating to evaluation feedback? 


Not 

important 

Fai rly 
important 

Important 

Very 

important 

(m = 28) 

Average* 

1 

6 

8 

12 

Domestic pressure for greater accountability 

3.0 

8 

10 

5 

3 

Pressure within recipient countries for 
greater aid accountability 

2.0 

2 

5 

10 

7 

The move towards participatory monitoring 
and evaluation (which raises questions of 
accountability and feedback to whom?) 

2.8 

1 

3 

9 

13 

The changing way development aid is 
delivered (the move towards sector and country- 
wide approaches - a trend which makes the 
evaluation process, and feedback from it, more 
complex) 

3.2 

2 

9 

7 

8 

The new opportunities for sharing evaluation 
feedback opening up thanks to the spread of 
the internet and other electronic 
communication channels 

2.7 

1 

10 

7 

8 

The increasing problem of "information 
overload" among agency staff 

2.7 


(n = 28) 


•averages are calculated by taking 4 = very important 3 = important 2 = fairly important 1 = not important. 

A lack of response is interpreted as implying a ranking of "not important". 

Norway’s answers could not be included as they did not give a ranking, only a tick. 

Selection of key comments and issues raised: 

• Particularly this section seems "biased" towards "aid delivery' which conflicts with a 
bank's requirement (at least EBRD's) to remain, on a sustainable basis and seen over- 
all its portfolio, a financially viable institution committed to observe sound banking 
principles. Again, ultimately the question needs to be addressed to the fund (capital, 
aid budget) providers of how much they (the governments, parliamentarians) are will- 
ing to consider as appropriate for executing evaluation functions (EBRD). 

• Concerning participatory monitoring and evaluation and the changing way develop- 
ment aid is delivered one could say that there is much talking but not so much acting 

(Sweden) 

• Pressure from recipient countries at the moment is not a major factor, but more needs 
to be done to create pressure - perhaps by building an evaluation culture in govern- 
ments and among officials - and for civil society to press for more accountability. Par- 
ticipatory M&E may have an important role to play in this (UK). 

• The (German) public seems to be less interested in scientific analysis and evaluation, 
but in more practical and stakeholder-based information (Germany-KfW). 
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• Accepting that in a climate of shrinking resources and greater accountability, the pres- 
sure to demonstrate the effectiveness of activities will increase, the Agency is pursuing 
the issues of performance information and monitoring, quality assurance, evaluation 
and feedback with extreme seriousness (Australia). 

• The increasing importance attached to the policy priorities related to poverty reduc- 
tion, environment, democracy and human rights has given rise to the need for evaluations 
in these areas both as overriding issues and as themes for separate evaluations 2. Low 
scoring for "pressure within recipient country” does not mean that it would not be 
important, but to our knowledge it has hardly ever happened (Finland). 
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Annex 3 

Evaluation Products 


Evaluation products have improved substantially over the last ten years. Agencies are 
paying more attention to the need to produce short and manageable summaries for a wider 
audience, in addition to the more detailed technical reports. Documents intended for public 
circulation have also improved in terms of "product branding ', with better quality design and 
improved presentation. In addition, a review of some of the agencies that participated* in 
this study shows that a wide range of products are being produced - targeted at different 
audiences, internally and externally. 


Summaries 

Most agencies have adopted the approach of producing short (4-6 page) summaries of 
specific evaluations (DFID - EvSwms, WorM Bank - Precis , DANIDA). For some agencies, the 
summary is up to 50 pages long ( Netherlands ). In some case the summary is not issued sepa- 
rately, put as a part of the main report (EC inventory, BMZ Programme Report). 


Periodicals and newsletters 

Some agencies now produce short newsletters that draw on more than one evaluation 
report, offer updates on on-going evaluations and/or draw out specific lessons learnt (OED 
R each, Evaluation Study Updates, SIDA — Evaluation Newsletter, CIDA Performance Newsletter). The EC 
Feedback newsletter is an internal document that summarises current or recent evaluations, 
and gives contact details of the evaluator and a web-site address for the full report, where 
appropriate. 


Full reports 

Evaluation reports range from the inclusive and technical to the subject specific and 
eclectic. Some agencies have published annexes and sub-reports separately (Netherlands and 
SIDA). KfW produces a number of versions of the same report - an internal version for the 
Ministry, an external report for specialists and professions and a report for project partners. 
Many agencies also produce an annual report detailing all evaluation studies to date and 
evaluation planned (SIDA). 


* Note that the information given here is based on the samples that were sent to the EFELA team, and is 
not intended to be comprehensive. 
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Other products 

The Netherlands produces two page press releases for completed evaluation. DANIDA 
has produced a video on one of its programme activities in Bhutan. The World Bank and DFID 
both produce development magazines aimed at a wider audience. The World Bank's OED 
department also provides a help desk, which can respond to internal and external informa- 
tion needs. The OED's information pamphlet details all of the services on offer and informa- 
tion on the web based resources. 

Dealing with language 

English is the dominant language of all the evaluation material, however the need to 
communicate more widely has been approached in a variety of ways. CIDA's performance 
newsletter is bilingual, SIDA's is published separately in two languages. OED has some trans- 
lations available on the web. SIDA has published its full report on Vietnamese. DANIDA publish 
a separate, booklet-sized Danish version of the full report for the domestic public. 
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Annex 4 

Review of Evaluation Web Resources 

A survey of development agencies web resources, and the comments received in 
response to the efela Questionnaire, make it clear that: 

• Agencies are at widely different stages with the development of their websites, both 
interna] and external. 

• Agencies place differing emphasis on Internet based dissemination in relation to 
other evaluation dissemination products. 

However, all agencies see the Internet as playing at least some role in their dissemination 
strategy. What follows is a preliminary review of some of the issues, techniques and concerns that 
have emerged across all DAC agencies. 


How to get there 

Mosl agencies have an area of their site that specifically hosts information on evaluation. How easy it is 
to get to this varies, however. 

The simplest way is to have links from the home page, clearly marked, that take the user 
directly to the evaluation section of the site, e.g. Asian Development Bank, World Bank. 

Frequently evaluation resources have to be accessed by first clicking on another link on 
the home page. This can be confusing if it is not obvious where to go, as the path is not stan- 
dard. Some examples of the buttons that have to be clicked to get to evaluation feedback 
are “Publications" (AusAid), “Information Resource Centre" (AFDB), “Areas of Work" (SIDA), 
"What we Do" (DFID). USAID is particularly complicated, with no link from the main page and 
the evaluation department under a different root directory. 


Format and accessibility 

Agencies have taken different approaches on the format in which they present summaries and full docu- 
mentation. This has implications in terms of ease of access. 

Many agencies have used pdf formats to put full reports on the web. Only a few use Word 
format (IADB). Agencies that have decided to post summaries on the web with links to the 
full document, make it a lot simpler to see which documents are relevant. Ireland, for exam- 
ple, has summaries in html format, which makes it easier to browse through and select which 
reports to download. This is much more useful than including summaries as a part of the main 
pdf file (GTZ). USAID is one example where summaries are provided separately as small pdf 
files which have to be downloaded. 1 l3 l 
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Finding relevant information 

The usefulness of the site depends not only on the extent and type of information to be found there, but 
also on the ease of finding relevant information. 

Some evaluation sites seem to be suffering from ever lengthening unclassified lists, so 
accessing information can be difficult (AusAid). Methods such as chronological listing 
(DANIDA) are fairly limiting when searching for a specific type of information. Finding a way 
out of this either means classifying reports under headings, or using a search engine function 
that is dedicated to the evaluation section. This is a lot more effective that relying on a search 
engine for the whole site. Good examples of this have been developed by Asian Development 
Bank and the DAC. 
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